Building a farm data and analytics platform that keeps things private

CAST is made up of three Cornell farms: the Cornell University Ruminant Center, the Cornell Teaching Dairy Barn and Musgrave Research Farm. Together, they serve as models of the U.S agricultural economy, and CAST’s interdisciplinary team of researchers is using them to create an ecosystem of networked technologies and techniques for the 21st-century farm.

A key part of CAST’s vision is to create a data infrastructure that automates the collection of data from disparate farm sensors — for example, biometric data from wearable sensors on livestock or environmental data from gas sensors at the manure pit — then stores it in a database and allows for AI-generated analytics. 

Shen is contributing to this aim by building an automated pipeline for the data streams. “We are building a scalable data backbone for CAST that would help researchers and collaborators utilize farm data more effectively,” he explained.

The resulting database will be both local at the farm and centralized in the cloud. “We do this because we may have many farms involved,” said Weatherspoon, who is also CAST associate director and co-principal investigator. “That’s the case right now with CAST, which has three farms, so the data is coming from multiple locations.”

The aggregation of data from many farms is important because predictive analytics are more accurate the larger the database they pull from. “Early studies have shown that if I just make predictions off my own farm data, that’s not as good as if I make predictions based off of the data from ten farms,” said Weatherspoon. “More situations will come up across ten farms than just on my own farm. So participating in a cooperative is better than going it alone.”

But if a farm is participating in a data cooperative, how can the privacy of each individual farm be protected? “We have to balance the privacy of the farmer with the ability to create advanced analytics or predictive machine learning models that require data from many sources,” Weatherspoon said. “The farmer might not want the insurance company or the neighbor down the road to know fine, granular details about what’s going on at the farm. So we have to obscure it in some way.”

Another of Weatherspoon’s graduate students, Salman Abid PhD’29, is tackling farmer privacy head on. Using CAST data as a model, Abid is bringing privacy guarantees to sensitive information captured on-farm, using techniques such as federated learning (which trains a shared AI model on multiple centralized devices, thus keeping raw data private) and federated analytics (in which the data remains where it is and the query — the code written by the analyst — is sent to the data).

"CAST is the farm of the future, and in that farm of the future we want farmers to understand what’s going on with their own data and have more control over it.”  

- Hakim Weatherspoon

“These techniques let farmers keep raw data on-premises for the entire duration of the process,” Abid said. “No raw or unprocessed data leaves the farm, nor is it seen in its original form by the model provider at any time.”

Abid is also figuring out ways to apply differential privacy to the CAST data. In this technique, data points are modified by a small degree so that the data is slightly different from what the user contributed but still achieves similar results when used for analytics or machine learning.

Ultimately, Weatherspoon and his colleagues envision the data analytics platform created for CAST as the basis for an infrastructure-scale project that could be used with any farm. “All farms are collecting data,” Weatherspoon said. “They all want to make use of that data and they’re all concerned about privacy and also about who profits off their data. CAST is the farm of the future, and in that farm of the future we want farmers to understand what’s going on with their own data and have more control over it.” 

Jackie Swift is the communications specialist for the Cornell CALS Department of Animal Science and the communications manager for CAST.