Rowe IT enables Met Office customers to access subsetted, global atmospheric model data
In October 2019 Rowe IT were engaged by the Met Office to define and build a capability to allow consumers and corporates to access Global high-resolution atmospheric model forecasts – a long defined goal for the Met Office.
Fast forward ten months and following a short Private Beta, the service is now operational, live and publicly available as a Public Beta, with a small but growing client base from across the globe. A major success for both Rowe IT and the Met Office.
Why did the Met Office want to publicly expose a data subsetting service?
Whilst the Met Office generates revenue from its activities, it is partly funded by the UK government. As part of that funding the Met Office commits to make its ‘public task data’ freely available over a level playing field.
The traditional route to market for the Met Office was via a Data Provisioning Team agreeing contractual terms with their customers. Whilst this channel is still important for large data users there was a desire to introduce a ‘self-help’ service for SME corporates and individual consumers to increase the reach of the Met Office into new markets at a lower delivery cost.
The Increasing volume and sheer scale of data generated (Terabytes per day) as model resolution increases is a real barrier to consumption. Subsetting data into chunks was required to allow the user to refine their model data requests by region and parameter data, amongst others, so the data volumes (and cost) are much more manageable.
What did Rowe IT deliver?
Embedded within all aspects of the project, Rowe IT delivered:
- the architecture
- user research and user experience (UX) design
- the technical solution
Rowe IT defined a micro-services architecture to interface to upstream and downstream systems within the existing Met Office infrastructure. The platform was designed to deal with high volumes of concurrent model data requests with delivery of terabytes of data daily, within a high availability, resilient design.
In order to inform the project, Rowe IT undertook user research through interactive interview sessions with current and potential Met Office customers. This was employed to determine their business needs and shape the user experience. Optimising user experience in terms of functionality and presentation, for users of varying technical and meteorological expertise, was a particular challenge.
The Rowe IT development team responsible for the technical solution delivered the underpinning backend services, GUI presentation, reporting and analytics across the service. The Cloud instances are based around a series of CI/CD pipelines to ensure continuous deployment and testing across multiple environments. The primary development language is Java with services deployed as multiple container instances. Cloud infrastructure is created and maintained using Terraform.
How did Rowe IT deliver?
The Rowe IT ethos is to always deliver as ‘one team’ with the customer. Rowe IT worked closely with Met Office Enterprise Architects, business teams, marketing and Programme/Project Management to ensure Met Office aspirations were clearly defined with a shared vision across the project.
All development was managed using an agile methodology based on two weekly sprints with clearly defined user stories and acceptance criteria. Transparency of status and issues was aided by use of Atlassian JIRA for sprint and task management and Confluence for shared documentation and designs. End of sprint demos ensured communication to a wider Met Office audience and concentration on incremental, deployable business value each sprint.
Early on, Rowe IT was heavily involved in the definition of a Minimum Viable Product (MVP), based upon user research and Met Office goals. The MVP definition was pivotal in focussing the mind on the critical objectives and ensuring business value was delivered as early as possible in the project.
What does the future hold?
The service has now been running in Public Beta since mid-August with few issues and a steadily growing customer base, given the initial soft launch. An important forward step will be to migrate existing Met Office customers away from legacy systems onto the new subsetting service.
The aim of the Public Beta phase is to open up the service to a wider number of customers, to gain feedback and drive further developments and improvements to the service ahead of its full launch with enhanced functionality in early 2021. As you would expect, things never stand still, and development is already underway delivering incremental improvements to current processes and software. Releases have already been delivered to improve invoicing processes and analytics as well as addressing backlog items not included within the MVP definition. In addition, the Met Office have ambitious plans to expand the forecast data available from UK and ensemble models and make new products, such as radar and lightning, available.