Handling vast amounts of unstructured and multimodal data – from images and videos to audio files and documents – has always been a significant challenge for data analytics platforms. Fortunately, a powerful new capability is transforming how organizations manage and analyze these complex data types within their familiar data warehouse environment.
This innovative feature allows you to register external objects, stored in cloud storage (like Google Cloud Storage), directly within your data warehouse. Instead of moving or transforming this large data, you create references – think of them as intelligent pointers – within structured tables. This capability, known as ObjectRef in BigQuery, unlocks the ability to query and process external data seamlessly alongside your structured data.
A primary advantage is the ability to work with unstructured and multimodal data without complex Extract, Transform, Load (ETL) processes. You can keep your data where it is, reducing data movement, complexity, and cost. Performance is also supercharged. By referencing external objects directly, you leverage the scalability and processing power of the data warehouse engine for metadata queries, and efficiently pass data to specialized functions for processing.
Perhaps the most exciting aspect is the integration with Artificial Intelligence and Machine Learning capabilities. With ObjectRef, you can easily apply pre-trained models or custom ML models (via BigQuery ML or integration with platforms like Vertex AI) directly to your external objects referenced within your tables. Imagine running image recognition on millions of product photos stored in cloud storage, or transcribing audio files – all initiated and managed from your data warehouse interface. This opens up entirely new avenues for intelligent data analysis on rich, unstructured datasets.
Beyond analytics and AI, ObjectRef simplifies data governance by providing a centralized point of access and metadata for both structured and unstructured assets. In essence, this feature bridges the gap between traditional structured data analysis and the world of unstructured and multimodal data, enabling powerful new insights and applications directly within your data analytics platform. It’s a significant step forward in handling the complexity of modern data landscapes and supercharging AI-driven analytics.
Source: https://cloud.google.com/blog/products/data-analytics/new-objectref-data-type-brings-unstructured-data-into-bigquery/
