# Batch Create Source: https://docs.deasylabs.com/api-reference/data-connectors/batch-create /deasy-openapi.yml post /data_connector/create_batch Create multiple data connectors in a single call. Useful for multi-site SharePoint connections where each site is a separate connector. # Batch Delete Source: https://docs.deasylabs.com/api-reference/data-connectors/batch-delete /deasy-openapi.yml post /data_connector/delete_batch Delete multiple data connectors in a single call. Useful for removing all connectors in a multi-site group. # Create Source: https://docs.deasylabs.com/api-reference/data-connectors/create /deasy-openapi.yml post /data_connector/create Create a new vector database (VDB) connector. ## Request Body | Field | Type | Description | |-------------------|---------------|--------------------------------------------------| | `connector_name` | `str` | The profile name of the data connector to create.| | `connector_body` | `VDBConfigs` | Configuration object for the VDB connector. | ## Response - **200**: VDB connector created successfully - Returns: `{ "profile_id": str }` - **500**: Internal Server Error (e.g., failed to store secret) ## Example ```json { "connector_name": "my-pinecone-db", "connector_body": { "vector_db_type": "pinecone", "api_key": "your-api-key", "environment": "us-east-1-aws", "index_name": "my-index" } } ``` # Delete Source: https://docs.deasylabs.com/api-reference/data-connectors/delete /deasy-openapi.yml post /data_connector/delete Delete a data connector and all associated data. This operation will: - Delete all vector data stored in the VDB - Remove the connector configuration - Clean up connection sharing metadata if enabled ## Request Body | Field | Type | Description | |-------------------|--------|--------------------------------------------------| | `connector_name` | `str` | The profile name of the data connector to delete.| ## Response - **200**: VDB connector deleted successfully - Returns: `{ "profile_id": str }` - **404**: Connector not found - **500**: Internal Server Error (e.g., failed to store secret) ## Example ```json { "connector_name": "my-pinecone-db" } ``` # Get Document Text Source: https://docs.deasylabs.com/api-reference/data-connectors/get-document-text /deasy-openapi.yml post /data/document_text Retrieve the raw text content for specified documents from the vector database # Get Ocr Page Text Source: https://docs.deasylabs.com/api-reference/data-connectors/get-ocr-page-text /deasy-openapi.yml post /get_ocr_page_text # Ingest Source: https://docs.deasylabs.com/api-reference/data-connectors/ingest /deasy-openapi.yml post /ocr/ingest Process documents with OCR and ingest them into Unstructured. This endpoint performs optical character recognition on documents and stores the extracted data. ## Request Body | Field | Type | Description | |--------------------------|-----------------|------------------------------------------------------------------------------| | `data_connector_name` | `str` | Name of the data connector to use. | | `file_names` | `List[str]` | Specific files to process. If omitted, processes all. | | `job_id` | `str` | Custom job ID for tracking. Auto-generated if not provided. | | `clean_up_out_of_sync` | `bool` | Remove files from VDB not in source. Default: `true`. | | `file_count_to_run` | `int` | Limit number of files to process. | | `use_llm` | `bool` | Use LLM for enhanced extraction. Default: `false`. | | `extract_compute_method` | `str` | Compute backend: `argo` (default), `spark`, or `local`. Acts as discriminator. | | `compute_configuration` | `object` | Backend-specific overrides. For `spark`: `executor_instances`, `executor_cores`, `task_cpus`, `executor_memory`, `executor_memory_overhead`, `driver_memory`. Empty object for `argo`/`local`. | ## Response - **200**: OCR job started successfully - Returns: `{ "message": str, "job_id": str }` - **400**: Bad Request (e.g., invalid data connector, unsupported VDB type) - **500**: Internal Server Error ## Example ```json { "data_connector_name": "my-documents", "use_llm": true, "clean_up_out_of_sync": true, "file_count_to_run": 100, "extract_compute_method": "spark", "compute_configuration": { "executor_instances": 8, "executor_memory": "4g" } } ``` # List Source: https://docs.deasylabs.com/api-reference/data-connectors/list /deasy-openapi.yml post /data_connector/list List all vector database (VDB) connectors for the authenticated user. ## Response - **200**: Successfully retrieved list of VDB connectors - Returns: `{ "connectors": VDBConfigDict }` ## Example Response ```json { "connectors": { "my-pinecone-db": { "vector_db_type": "pinecone", "environment": "us-east-1-aws", "index_name": "my-index" }, "my-qdrant-db": { "vector_db_type": "qdrant", "url": "https://qdrant.example.com" } } } ``` # List Ingested Data Source: https://docs.deasylabs.com/api-reference/data-connectors/list-ingested-data /deasy-openapi.yml post /data/metadata/list Retrieve metadata from documents ingested into Unstructured. ## Request Body | Field | Type | Description | |----------------------------|---------------------|-------------------------------------------------------| | `data_connector_name` | `str` | Name of the data connector to query. | | `group_by` | `str` | Group results by `file` or `node`. Default: `file`. | | `metadata_keys` | `List[str]` | Specific metadata keys to retrieve. | | `file_names` | `List[str]` | Filter by specific file names. | | `point_ids` | `List[str]` | Filter by specific node/point IDs. | | `metadata_key_filters` | `List[str]` | Filter by presence of metadata keys. | | `metadata_value_filters` | `Dict[str, List]` | Filter by specific metadata values. | | `full_text_filters` | `List[str]` | Full-text search filters. | | `limit` | `int` | Maximum number of results to return. | | `with_vectors` | `bool` | Include vector embeddings. Default: `false`. | ## Response - **200**: Successfully retrieved metadata - Returns: `{ "metadata": Dict }` - **500**: Internal Server Error (e.g., error fetching metadata) ## Example ```json { "data_connector_name": "my-documents", "group_by": "file", "metadata_keys": ["title", "author", "date"], "file_names": ["document1.pdf", "document2.pdf"], "limit": 100 } ``` # List Onedrive Sites Source: https://docs.deasylabs.com/api-reference/data-connectors/list-onedrive-sites /deasy-openapi.yml post /onedrive/list_sites List all OneDrive sites in the tenant (same Graph API as SharePoint). # List Paginated Source: https://docs.deasylabs.com/api-reference/data-connectors/list-paginated /deasy-openapi.yml post /data/list_paginated Retrieve a paginated list of files/nodes from the vector database - supports shared connections # List S3 Buckets Source: https://docs.deasylabs.com/api-reference/data-connectors/list-s3-buckets /deasy-openapi.yml post /s3/list_buckets List all S3 buckets accessible with the provided credentials. # List Sharepoint Sites Source: https://docs.deasylabs.com/api-reference/data-connectors/list-sharepoint-sites /deasy-openapi.yml post /sharepoint/list_sites List all SharePoint sites in the tenant. # List Source Files Source: https://docs.deasylabs.com/api-reference/data-connectors/list-source-files /deasy-openapi.yml post /ocr/list_source_files List files available for ingestion from a data source (S3, SharePoint). This endpoint queries the actual source storage (not the vector DB) to show files before they are ingested. Use this to enable selective file ingestion. ## Request Body | Field | Type | Description | |-----------------------|----------|--------------------------------------------------| | `data_connector_name` | `str` | Name of the data connector to query | | `prefix` | `str` | Optional path prefix to filter files | | `search_query` | `str` | Optional filename search (case-insensitive) | | `limit` | `int` | Max files to return (default: 1000, max: 10000) | | `offset` | `int` | Number of files to skip for pagination | ## Response - `files`: List of file objects with path, size, last_modified - `total_count`: Total number of matching files in the source - `next_offset`: Offset for next page (null if no more pages) - `selection_enabled`: False if total_count > 10000 (use full ingestion instead) # Ocr Sync Stats Source: https://docs.deasylabs.com/api-reference/data-connectors/ocr-sync-stats /deasy-openapi.yml post /ocr/sync_stats Get OCR sync statistics for multiple data connectors from cached database values. This endpoint only retrieves pre-calculated statistics from the database. If stats are older than 15 minutes, a background task is triggered to refresh them. # Set Active Connector Source: https://docs.deasylabs.com/api-reference/data-connectors/set-active-connector /deasy-openapi.yml post /connector/set_active_profile Set the active connector for the calling user. Authorization: The caller must either own the named connector or have it explicitly shared with them (directly or via one of their groups). An empty ``connector_name`` is accepted and clears the active connector. Attributes: connector_name: The profile name of the connector to set as active. connector_type: The type of connector to set as active, either "vdb" or "llm". # Update Source: https://docs.deasylabs.com/api-reference/data-connectors/update /deasy-openapi.yml post /data_connector/update Update an existing data connector configuration. ## Request Body | Field | Type | Description | |-------------------|---------------|--------------------------------------------------| | `connector_name` | `str` | The profile name of the data connector to update.| | `connector_body` | `VDBConfigs` | Updated configuration object for the VDB. | ## Response - **200**: VDB connector updated successfully - Returns: `{ "profile_id": str }` - **500**: Internal Server Error (e.g., failed to store secret) ## Example ```json { "connector_name": "my-pinecone-db", "connector_body": { "vector_db_type": "pinecone", "api_key": "updated-api-key", "environment": "us-west-1-aws", "index_name": "my-index" } } ``` # Check Dataslice Sync Endpoint Source: https://docs.deasylabs.com/api-reference/data-slices/check-dataslice-sync-endpoint /deasy-openapi.yml post /dataslice/sync_score Check the synchronization status of a specific data slice # Create Source: https://docs.deasylabs.com/api-reference/data-slices/create /deasy-openapi.yml post /dataslice/create Create a new data slice based on specified conditions. ## Request Body | Field | Type | Description | |-------------------|----------------|----------------------------------------------------------------------------| | `dataslice_name` | `str` | The name to assign to the newly created data slice. | | `condition` | `Optional[List[Dict]]` | (Legacy) List of file qualifying conditions. | | `condition_new` | `Optional[Any]` | New format: advanced file qualifying conditions. | | `description` | `Optional[str]` | Description of the data slice. | | `status` | `str` | The status of the data slice (e.g., active). | | `data_points` | `Optional[int]` | Number of data points in the data slice. | | `latest_taxonomy` | `Optional[Dict]` | Most recent taxonomy reference/metadata. | | `taxonomy_id` | `Optional[str]` | ID of the related taxonomy, if applicable. | | `data_connector_name` | `str` | Name of the data connector (vector database profile). | | `parent_dataslice_id` | `Optional[str]` | (Optional) Parent data slice ID, for lineage. | ## Response - **201**: Data slice created successfully - Returns: `{ "dataslice_id": "" }` - **500**: Internal Server Error. ## Example ```json { "dataslice_name": "Customer Purchases Q2", "condition": [ { "field": "created_date", "op": "gte", "value": "2024-04-01" } ], "description": "Purchase data from Q2 2024", "status": "active", "data_points": 4000, "data_connector_name": "prod-warehouse" } ``` # Delete Source: https://docs.deasylabs.com/api-reference/data-slices/delete /deasy-openapi.yml delete /dataslice/delete Delete a single data slice by its unique identifier. ## Request Body | Field | Type | Description | |-------------------|----------------|----------------------------------------------------------------------------| | `dataslice_id` | `str` | The unique identifier of the data slice to delete. | ### Response Returns a confirmation message of the successful deletion and any associated data. ```json { "message": "Data slice deleted successfully", "data": { // Details of deletion outcome } } ``` # Export Source: https://docs.deasylabs.com/api-reference/data-slices/export /deasy-openapi.yml post /dataslice/export/metadata Export metadata for a specific data slice or the entire data connector, in JSON or CSV format. ## Request Body | Field | Type | Description | |----------------------------|-----------------------|------------------------------------------------------------------------------------| | `data_connector_name` | `str` | Name of the data connector to export metadata from. | | `dataslice_id` | `Optional[str]` | ID of the dataslice to export metadata from. If omitted, all data is included. | | `export_file_level` | `bool` | `True` for file-level export, `False` for chunk-level export. | | `export_format` | `Optional[ExportFormat]` | Desired export format: `json` or `csv` (e.g., `"json"`, `"csv"`). | | `selected_metadata_fields` | `Optional[List[str]]` | List of metadata fields to include in the export. | ## Response - **200**: Metadata exported successfully - Returns: A streaming file response containing the exported metadata in the requested format. - **500**: Internal Server Error ## Example ```json { "data_connector_name": "example-connector", "dataslice_id": "abc123", "export_file_level": true, "export_format": "json", "selected_metadata_fields": ["field1", "field2"] } ``` # Export Dataslice2Vdb Source: https://docs.deasylabs.com/api-reference/data-slices/export-dataslice2vdb /deasy-openapi.yml post /dataslice/export/vdb Export metadata for a data slice to a target vector database # Get Dataslice File Count Source: https://docs.deasylabs.com/api-reference/data-slices/get-dataslice-file-count /deasy-openapi.yml post /dataslice/file_count Get count of files matching dataslice conditions or provided conditions Supports shared connections for Contributors and Admins. # Get Dataslice Metrics Endpoint Source: https://docs.deasylabs.com/api-reference/data-slices/get-dataslice-metrics-endpoint /deasy-openapi.yml post /dataslice/metrics Retrieve data slice metrics # List Source: https://docs.deasylabs.com/api-reference/data-slices/list /deasy-openapi.yml get /dataslice/list Retrieve all data slices associated with the currently authenticated user. ### Response Returns a JSON object containing a list of dataslices: ```json { "dataslices": [ { "dataslice_id": "string", "dataslice_name": "string", "description": "string", "status": "active", "data_points": 123, "latest_taxonomy": { /* taxonomy details */ }, "taxonomy_id": "string", "data_connector_name": "string", "parent_dataslice_id": "string or null" }, ... ] } ``` # Suggest Endpoint Source: https://docs.deasylabs.com/api-reference/data-slices/suggest-endpoint /deasy-openapi.yml post /dataslice/suggest Suggest a data slice # Create Ai Ready Slice Source: https://docs.deasylabs.com/api-reference/deduplication/create-ai-ready-slice /deasy-openapi.yml post /dedup/create_slice Create an AI-ready data slice by tagging duplicate files as redundant. 1. Ensures the 'Data Quality Status' tag exists 2. Tags non-representative duplicate files as 'redundant', but only when the caller chose to exclude 'redundant' (see ``exclude_statuses``) — tagging is a persistent side effect and must follow the caller's selection rather than running unconditionally 3. Creates a data slice with condition excluding the caller's chosen Data Quality Status values (see ``exclude_statuses``) # Dedup Preflight Source: https://docs.deasylabs.com/api-reference/deduplication/dedup-preflight /deasy-openapi.yml post /dedup/preflight Check tag coverage before running dedup. # Get Dedup Results Source: https://docs.deasylabs.com/api-reference/deduplication/get-dedup-results /deasy-openapi.yml post /dedup/results Retrieve detected duplicate groups. # Get Dedup Summary Source: https://docs.deasylabs.com/api-reference/deduplication/get-dedup-summary /deasy-openapi.yml post /dedup/summary Get summary statistics for dedup results. # Get Dedup Taxonomy Source: https://docs.deasylabs.com/api-reference/deduplication/get-dedup-taxonomy /deasy-openapi.yml get /dedup/taxonomy Return the OOB tag taxonomy for dedup: which tags are used for fingerprinting, date-based selection, and versioning. # Get Fingerprint Source: https://docs.deasylabs.com/api-reference/deduplication/get-fingerprint /deasy-openapi.yml post /dedup/fingerprint Get the computed fingerprint for a specific file. # Run Dedup Source: https://docs.deasylabs.com/api-reference/deduplication/run-dedup /deasy-openapi.yml post /dedup/run Run hash-based metadata deduplication on a data connector. # Create Source: https://docs.deasylabs.com/api-reference/destinations/create /deasy-openapi.yml post /destination/create Create a new destination connector for data export. ## Request Body | Field | Type | Description | |-------------------|-----------------------|--------------------------------------------------| | `connector_name` | `str` | The profile name of the destination to create. | | `connector_body` | `DestinationConfigs` | Configuration object for the destination. | ## Response - **200**: Destination created successfully - Returns: `{ "profile_id": str }` - **500**: Internal Server Error (e.g., failed to store secret) ## Example ```json { "connector_name": "my-destination", "connector_body": { "type": "s3", "config": { "bucket": "my-bucket", "region": "us-east-1" } } } ``` # Delete Source: https://docs.deasylabs.com/api-reference/destinations/delete /deasy-openapi.yml post /destination/delete Delete a destination connector. **Request Body** ```json { "connector_name": "my-s3-destination" } ``` | Field | Type | Description | |------------------|-------|-------------------------------------------------| | connector_name | str | The profile name of the destination to delete. | **Response** - 200: Destination deleted successfully - Returns: `{ "profile_id": str }` # Export Source: https://docs.deasylabs.com/api-reference/destinations/export /deasy-openapi.yml post /dataslice/export/metadata/destination Export dataslice metadata and/or nodes to a configured destination. This endpoint supports exporting data from a dataslice or data connector to a destination system. For small datasets (fewer than 100 files), the export runs synchronously. For larger datasets, the export is processed in the background and returns a tracker ID for monitoring progress. ## Request Body | Field | Type | Required | Description | |----------------------------|----------------------------|----------|---------------------------------------------------------------------------------------------------| | `dataslice_id` | `str` | No | The ID of the dataslice to export. Required if `data_connector_name` is not provided. | | `data_connector_name` | `str` | No | The name of the data connector to export from. Required if `dataslice_id` is not provided. | | `destination_name` | `str` | No | The name of the configured destination to export to. | | `export_tags` | `List[str]` | No | List of specific tags to export. If empty, all tags are exported. | | `export_level` | `ExportLevel` | No | Level of export: `"file"`, `"chunk"`, or `"both"` (default: `"both"`). | | `export_nodes` | `bool` | No | Whether to export node data (default: `true`). | | `export_metadata` | `bool` | No | Whether to export metadata (default: `true`). | | `metadata_format` | `MetadataStorageFormat` | No | Format for metadata storage: `"column_store"` or `"json_store"`. | ## Response - **200**: Export completed or initiated successfully - Returns: `ExportDatasliceDestinationResponse` with: - `message`: Status message - `success`: Number of successfully exported items - `failed`: Number of failed exports - `failed_items`: List of items that failed to export - `failed_tags`: List of tags that failed to export (if applicable) - `tracker_id`: Progress tracker ID for background exports (only for large datasets) - **400**: Bad Request (e.g., missing required parameters) - **500**: Internal Server Error ## Example Request ```json { "dataslice_id": "550e8400-e29b-41d4-a716-446655440000", "destination_name": "my-destination", "export_tags": ["contracts", "invoices"], "export_level": "both", "export_nodes": true, "export_metadata": true, "metadata_format": "column_store" } ``` ## Example Response (Synchronous) ```json { "message": "Export completed successfully", "success": 45, "failed": 0, "failed_items": [], "failed_tags": null, "tracker_id": null } ``` ## Example Response (Background Processing) ```json { "message": "Export started in background. Use tracker_id to monitor progress.", "success": 0, "failed": 0, "failed_items": [], "failed_tags": null, "tracker_id": "export-12345678-90ab-cdef-1234-567890abcdef" } ``` # List Source: https://docs.deasylabs.com/api-reference/destinations/list /deasy-openapi.yml post /destination/list List all destinations for the authenticated user. ## Response - **200**: Successfully retrieved list of destinations - Returns: `{ "connectors": DestinationConfigDict }` ## Example Response ```json { "connectors": { "my-s3-destination": { "type": "s3", "config": { "bucket": "my-bucket", "region": "us-east-1" } }, "my-webhook-destination": { "type": "webhook", "config": { "url": "https://api.example.com/webhook" } } } } ``` # API Introduction Source: https://docs.deasylabs.com/api-reference/introduction Authentication, conventions, and patterns shared by every endpoint The Deasy Labs REST API powers everything the platform does; the [Python SDK](/quickstart) wraps it one-to-one, so every endpoint here has a matching SDK method. Use this page for the conventions shared by all endpoints, then browse the resources in the sidebar. ## Base URL All requests go to your deployment's base URL. There is no default. ```text theme={"dark"} https://unstructured.your-company.com/rest/unstructured ``` ## Authentication Two methods are supported: | Method | Headers | Notes | | :------------ | :---------------------------------------------------------- | :----------------------------------------------- | | **Basic** | `Authorization: Basic ` | Username and password | | **API token** | `Authorization: Bearer ` plus `X-User-ID: ` | Tokens are long-lived and issued from the web UI | In the SDK, these map to `username`/`password` or `api_token`/`user_id`, with environment-variable fallbacks (`UNSTRUCTURED_USERNAME`, `UNSTRUCTURED_PASSWORD`, `UNSTRUCTURED_API_TOKEN`, `UNSTRUCTURED_USER_ID`, `UNSTRUCTURED_CLIENT_BASE_URL`). ## Background Jobs Long-running operations (ingestion, batch classification, versioning, large exports) return immediately and run in the background. Pass a `job_id` of your choosing, then poll Task Tracking: 1. Submit the job with a `job_id` (a UUID you generate). 2. Poll `POST /progress_tracker/task_status` with that id. 3. `status` moves through `in_progress` to `completed`, `failed`, or `aborted`, with `percent_complete` along the way. ## Pagination List endpoints that can return large results offer paginated variants (`/metadata/list_paginated`, `/data/list_paginated`) using `limit` and `offset`, returning `next_offset` until it is null. ## Errors Standard HTTP status codes: `400` bad request, `401` authentication, `403` permission, `404` not found, `409` conflict, `422` validation, `429` rate limit, `5xx` server. The SDK retries twice by default on retryable failures and raises typed exceptions per status. # Classify Source: https://docs.deasylabs.com/api-reference/metadata/classify /deasy-openapi.yml post /classify Classify specified files with the provided tags. To run sensitivity detection, use tags with a strategy of SENSITIVITY or a group which includes the magic word "Sensitivity" ## Request Body | Field | Type | Description | |-----------------------|-----------------------|------------------------------------------------------------------------------| | `data_connector_name` | `str` | Name of the data connector (vector database profile) to use for classification. | | `file_names` | `Optional[List[str]]` | Names of specific files to classify. | | `tag_names` | `Optional[List[str]]` | Names of tags to use for classification (if `tag_datas` not provided). | | `tag_datas` | `Optional[Dict]` | Tag data to use for classification. | | `overwrite` | `bool` | Whether to overwrite existing tags. Default: `false` | | `job_id` | `Optional[str]` | Custom job ID for tracking the classification task. | | `soft_run` | `bool` | If `true`, classification will not save to Deasy and will return results. Default: `false` | | `hierarchy_name` | `Optional[str]` | Name of the hierarchy/taxonomy to use (if `hierarchy_data` not provided). | | `hierarchy_data` | `Optional[Dict]` | Hierarchy/taxonomy data to use for classification. Default: `{}` | | `dataslice_id` | `Optional[str]` | ID of the dataslice to use for file filtering. | ## Response - **200**: Classification completed or job started successfully - Returns: `{ "message": str, "job_id": str, "results": Optional[Dict] }` - **500**: Internal Server Error ## Example ```json { "data_connector_name": "my-connector", "file_names": ["document1.pdf", "document2.pdf"], "tag_names": ["category", "sentiment"], "overwrite": false, "soft_run": false } ``` # Classify Bulk Source: https://docs.deasylabs.com/api-reference/metadata/classify-bulk /deasy-openapi.yml post /classify_bulk Classify all files in a data connector in batches with the provided tags. To run sensitivity detection, use tags with a strategy of SENSITIVITY or a group which includes the magic word "Sensitivity" ## Request Body | Field | Type | Description | |-----------------------|-----------------------|------------------------------------------------------------------------------| | `data_connector_name` | `str` | Name of the data connector (vector database profile) to use for classification. | | `total_data_sets` | `Optional[int]` | Total number of files to classify. | | `tag_names` | `Optional[List[str]]` | Names of tags to use for classification (if `tag_datas` not provided). | | `tag_datas` | `Optional[Dict]` | Tag data to use for classification. | | `overwrite` | `bool` | Whether to overwrite existing tags. Default: `false` | | `job_id` | `Optional[str]` | Custom job ID for tracking the classification task. | | `hierarchy_name` | `Optional[str]` | Name of the hierarchy/taxonomy to use (if `hierarchy_data` not provided). | | `hierarchy_data` | `Optional[Dict]` | Hierarchy/taxonomy data to use for classification. Default: `{}` | | `dataslice_id` | `Optional[str]` | ID of the dataslice to use for file filtering. | | `conditions` | `Optional[Condition]` | Conditions to use for file filtering. | ## Response - **200**: Classification job started successfully - Returns: `{ "message": str, "job_id": str }` - **500**: Internal Server Error ## Example ```json { "data_connector_name": "my-connector", "tag_names": ["category", "sentiment"], "overwrite": false, "dataslice_id": "abc123" } ``` # Delete Source: https://docs.deasylabs.com/api-reference/metadata/delete /deasy-openapi.yml post /metadata/delete Delete metadata for specified files, tags, and/or conditions. ## Request Body | Field | Type | Description | |-----------------------|---------------------|------------------------------------------------------------------------------| | `data_connector_name` | `str` | Name of the data connector (vector database profile) to delete metadata from.| | `file_names` | `Optional[List[str]]` | List of file names to delete metadata for. | | `tags` | `Optional[List[str]]` | List of tags to filter which metadata to delete. | | `conditions` | `Optional[Condition]` | Additional conditions to filter which metadata to delete. | ## Response - **200**: Metadata deleted successfully - Returns: `{ "chunk_deleted_count": int, "file_deleted_count": int }` - **500**: Internal Server Error ## Example ```json { "data_connector_name": "my-connector", "file_names": ["document1.pdf", "document2.pdf"], "tags": ["archived", "old"] } ``` # Delete Standardization Source: https://docs.deasylabs.com/api-reference/metadata/delete-standardization /deasy-openapi.yml post /metadata/standardize/delete Delete a standardization mapping # Delete Tag Values Source: https://docs.deasylabs.com/api-reference/metadata/delete-tag-values /deasy-openapi.yml post /metadata/delete/values Delete specific values from a tag across all files. Accepts tag_name (str), values (List[str]), data_connector_name (str), and optional dataslice_id (str). # Get Evidence Source: https://docs.deasylabs.com/api-reference/metadata/get-evidence /deasy-openapi.yml post /metadata/get_evidence Retrieve evidence for specified file and tag - supports shared connections # Get Filtered Metadata Source: https://docs.deasylabs.com/api-reference/metadata/get-filtered-metadata /deasy-openapi.yml post /metadata/filtered Get paginated filtered metadata based on conditions # Get Unique Tags Source: https://docs.deasylabs.com/api-reference/metadata/get-unique-tags /deasy-openapi.yml post /metadata/get_unique_tags Get a list of unique tags that have been extracted - supports shared connections # Insert Standardization Source: https://docs.deasylabs.com/api-reference/metadata/insert-standardization /deasy-openapi.yml post /metadata/standardize/insert Insert or update a standardization mapping for a tag # List Source: https://docs.deasylabs.com/api-reference/metadata/list /deasy-openapi.yml post /metadata/list Get filtered metadata based on conditions. Supports shared connections. ## Request Body | Field | Type | Description | |-----------------------|-----------------------|------------------------------------------------------------------------------| | `data_connector_name` | `str` | Name of the data connector (vector database profile). | | `dataslice_id` | `Optional[str]` | ID of the dataslice to get files from. | | `tag_names` | `Optional[List[str]]` | List of tag names to include in the metadata. | | `include_chunk_level` | `Optional[bool]` | Whether to include chunk-level metadata. Default: `true` | | `file_names` | `Optional[List[str]]` | List of specific file names to include in the metadata. | | `chunk_ids` | `Optional[List[str]]` | List of specific chunk IDs to include in the metadata. | | `include_last_updated`| `Optional[bool]` | Whether to include the last updated timestamp. Default: `false` | ## Response - **200**: Metadata retrieved successfully - Without `chunk_ids`: Returns metadata grouped by filename, tag, and level `{filename: {tag_id: {chunk_level: {chunk_id: metadata}, file_level: metadata}}}` - With `chunk_ids`: Returns metadata by chunk ID `{chunk_id: {metadata}}` - **500**: Internal Server Error ## Example ```json { "data_connector_name": "my-connector", "dataslice_id": "abc123", "tag_names": ["category", "date"], "include_chunk_level": true, "file_names": ["document1.pdf", "document2.pdf"] } ``` # List Chunk Metadata Source: https://docs.deasylabs.com/api-reference/metadata/list-chunk-metadata /deasy-openapi.yml post /metadata/chunk/list Get metadata for specified files and tags # List Paginated Source: https://docs.deasylabs.com/api-reference/metadata/list-paginated /deasy-openapi.yml post /metadata/list_paginated Get paginated filtered metadata based on conditions. ## Request Body | Field | Type | Description | |-----------------------|-----------------------|------------------------------------------------------------------------------| | `data_connector_name` | `str` | Name of the data connector (vector database profile). | | `dataslice_id` | `Optional[str]` | ID of the dataslice to get files from. | | `tag_names` | `Optional[List[str]]` | List of tag names to include in the metadata. | | `include_chunk_level` | `Optional[bool]` | Whether to include chunk-level metadata. Default: `true` | | `offset` | `Optional[int]` | Pagination offset to start from. Default: `0` | | `limit` | `Optional[int]` | Maximum number of metadata items to return. Default: `50` | | `include_last_updated`| `Optional[bool]` | Whether to include the last updated timestamp. Default: `false` | ## Response - **200**: Metadata retrieved successfully - Returns: `{ "metadata": Deasy_Metadata, "next_offset": Optional[int] }` - **500**: Internal Server Error ## Example ```json { "data_connector_name": "my-connector", "dataslice_id": "abc123", "tag_names": ["category", "date"], "include_chunk_level": true, "offset": 0, "limit": 50 } ``` # List Standardizations Source: https://docs.deasylabs.com/api-reference/metadata/list-standardizations /deasy-openapi.yml post /metadata/standardize/list Retrieve all standardization mappings for a user # Standardize Metadata Source: https://docs.deasylabs.com/api-reference/metadata/standardize-metadata /deasy-openapi.yml post /metadata/standardization_suggest Standardize metadata values using LLM — supports multiple tags with optional auto-apply. # Standardize Metadata Bulk Source: https://docs.deasylabs.com/api-reference/metadata/standardize-metadata-bulk /deasy-openapi.yml post /metadata/standardization_bulk Start bulk standardization as a background task and return job ID immediately. # Standardize Metadata Db Source: https://docs.deasylabs.com/api-reference/metadata/standardize-metadata-db /deasy-openapi.yml post /metadata/standardization_db Apply standardization mapping to metadata values in database # Upsert Source: https://docs.deasylabs.com/api-reference/metadata/upsert /deasy-openapi.yml post /metadata/upsert Insert or update metadata for files and tags. ## Request Body | Field | Type | Description | |-----------------------|-----------------------|------------------------------------------------------------------------------| | `data_connector_name` | `Optional[str]` | Name of the data connector (vector database profile). | | `dataslice_id` | `Optional[str]` | ID of the dataslice to upsert metadata for. | | `metadata` | `Deasy_Metadata` | Metadata to upsert in the form `{file_name: {tag_name: tag_value}}`. | ## Response - **200**: Metadata upserted successfully - Returns: `{ "success": bool }` - **500**: Internal Server Error ## Example ```json { "data_connector_name": "my-connector", "metadata": { "document1.pdf": { "category": "invoice", "date": "2024-01-15" }, "document2.pdf": { "category": "receipt", "status": "processed" } } } ``` # Create Project Source: https://docs.deasylabs.com/api-reference/projects/create-project /deasy-openapi.yml post /projects/create # Delete Project Source: https://docs.deasylabs.com/api-reference/projects/delete-project /deasy-openapi.yml delete /projects/{project_id} # Get Project Source: https://docs.deasylabs.com/api-reference/projects/get-project /deasy-openapi.yml get /projects/{project_id} # Get Project Sensitive Data Source: https://docs.deasylabs.com/api-reference/projects/get-project-sensitive-data /deasy-openapi.yml get /projects/{project_id}/sensitive-data # List Projects Source: https://docs.deasylabs.com/api-reference/projects/list-projects /deasy-openapi.yml post /projects/list # Update Project Source: https://docs.deasylabs.com/api-reference/projects/update-project /deasy-openapi.yml put /projects/{project_id} # Bulk Upsert Tag Source: https://docs.deasylabs.com/api-reference/tags/bulk-upsert-tag /deasy-openapi.yml post /tags/upsert_bulk Bulk upsert tags Attributes: tag_data: The tag data to upsert the tag with. # Create Tag Route Source: https://docs.deasylabs.com/api-reference/tags/create-tag-route /deasy-openapi.yml post /tags/create Create a new tag Attributes: tag_data: The tag data to create the tag with. # Delete Source: https://docs.deasylabs.com/api-reference/tags/delete /deasy-openapi.yml delete /tags/delete Delete a tag by name. Tag must not be in use by any graphs or dataslices. ## Query Parameters | Parameter | Type | Description | |------------|-------|--------------------------------------| | `tag_name` | `str` | Name of the tag to delete. | ## Response - **200**: Tag deleted successfully - Returns: `{ "tag_name": str }` - **500**: Internal Server Error (e.g., tag still in use) ## Example ``` DELETE /tags/delete?tag_name=category ``` # Evaluate Test Cases Source: https://docs.deasylabs.com/api-reference/tags/evaluate-test-cases /deasy-openapi.yml post /evaluate_test_cases Evaluate test cases against multiple regex patterns. This route allows users to validate regex patterns using Python's regex engine, which is important because Python regex and JavaScript regex have differences. Tests each case against all patterns and returns whether it matched. A test case is considered to match if ANY of the provided patterns match it. Args: patterns: List of regex patterns to test (with description and optional output_value) test_cases: List of test cases with text, expected match behavior, and category Returns: EvaluateTestCasesResponse with evaluated test cases showing actual_match results # Generate Test Cases Source: https://docs.deasylabs.com/api-reference/tags/generate-test-cases /deasy-openapi.yml post /generate_test_cases Generate test cases for regex pattern validation. Creates both positive (should match) and negative (should NOT match) test cases to help users validate their regex patterns. Attributes: pattern_descriptions: List of descriptions of what patterns should match. existing_patterns: Optional existing regex patterns to test against. # Get Default Tags Route Source: https://docs.deasylabs.com/api-reference/tags/get-default-tags-route /deasy-openapi.yml get /tags/default Return the current OOB default tag definitions (name + available_values). # Get Searchability Scores Source: https://docs.deasylabs.com/api-reference/tags/get-searchability-scores /deasy-openapi.yml get /tags/searchability_scores Get stored searchability scores for tags in a dataset context. ## Query Parameters | Parameter | Type | Required | Description | |---------------------|----------------|----------|------------------------------------------------| | `data_connector_name`| `str` | Yes | Name of the data connector/dataset | | `dataslice_id` | `str` | No | Optional dataslice ID for filtered scores | | `tag_names` | `List[str]` | No | Optional list of specific tag names | ## Response Returns a simple dictionary mapping tag names to their searchability scores: ```json { "Invoice Number": 0.85, "Date": 0.92, "Vendor": 0.78 } ``` Frontend can merge this with tag definitions client-side for optimal performance. # Get Sensitivity Default Tags Route Source: https://docs.deasylabs.com/api-reference/tags/get-sensitivity-default-tags-route /deasy-openapi.yml get /tags/sensitivity_default Return the current OOB sensitivity tag definitions (name + available_values). # Get Tag Scores Source: https://docs.deasylabs.com/api-reference/tags/get-tag-scores /deasy-openapi.yml post /tags/scores Get tag scores # List Source: https://docs.deasylabs.com/api-reference/tags/list /deasy-openapi.yml get /tags/list List all tags for the authenticated user. ## Response - **200**: Tags retrieved successfully - Returns: `{ "tags": List[Tag] }` - **500**: Internal Server Error ## Example Response ```json { "tags": [ { "name": "category", "description": "Document category", "type": "select", "available_values": ["invoice", "receipt"] }, { "name": "date", "description": "Document date", "type": "date" } ] } ``` # Suggest Description Source: https://docs.deasylabs.com/api-reference/tags/suggest-description /deasy-openapi.yml post /suggest_description Suggest a description for a tag based on context and vector DB content Attributes: data_connector_name: The name of the vdb profile to include in the dataslice. tag_name: The name of the tag to suggest a description for. context: The context to suggest a description for the tag. current_description: The current description of the tag. available_values: The available values for the tag. # Suggest Extraction Strategy Source: https://docs.deasylabs.com/api-reference/tags/suggest-extraction-strategy /deasy-openapi.yml post /suggest_strategy Determine the optimal extraction strategy for a tag based on its name and description. Returns either 'LLM' (AI-based extraction) or 'REGEX' (pattern-based extraction). ## When to use each strategy **REGEX** - Use when: - Values have consistent, predictable formatting (dates, IDs, phone numbers, emails) - Known exact patterns that repeat predictably - Structured data with fixed delimiters or formats **LLM** - Use when: - Semantic understanding is required - Values depend on interpreting content, not matching formats - Classifications, categorizations, sentiment, summaries ## Request Body | Field | Type | Description | |--------------------|----------------|------------------------------------------------| | `tag_name` | `str` | The name of the tag | | `tag_description` | `str` | Description of what the tag captures | | `available_values` | `List[str]` | Optional. Example values to help determine strategy | ## Response - **200**: Strategy recommendation returned successfully - Returns: `{ "strategy": "LLM" }` or `{ "strategy": "REGEX" }` - **500**: Internal Server Error # Suggest Patterns Source: https://docs.deasylabs.com/api-reference/tags/suggest-patterns /deasy-openapi.yml post /suggest_regex Generate a regex pattern based on a natural language description using AI. ## Request Body | Field | Type | Description | |--------------------|----------------|------------------------------------------------------------------------------------------| | `description` | `str` | Natural language description of what to match (e.g., "phone numbers in format (XXX) XXX-XXXX"). | | `examples` | `List[str]` | Optional. List of example strings that should match the pattern. | ## Response - **200**: Regex pattern generated successfully - Returns: `{ "regex": str }` - **400**: Bad Request (e.g., invalid request data) - **500**: Internal Server Error ## Example ```json { "description": "phone numbers in format (XXX) XXX-XXXX", "examples": [ "(123) 456-7890", "(555) 123-4567" ] } ``` ## Example Response ```json { "regex": "\(\d{3}\) \d{3}-\d{4}", } ``` # Update Tag Route Source: https://docs.deasylabs.com/api-reference/tags/update-tag-route /deasy-openapi.yml put /tags/update Update an existing tag Attributes: tag_data: The tag data to update the tag with. # Update Tag Searchability Scores Source: https://docs.deasylabs.com/api-reference/tags/update-tag-searchability-scores /deasy-openapi.yml post /tags/update_searchability_scores Get tag searchability scores # Upsert Source: https://docs.deasylabs.com/api-reference/tags/upsert /deasy-openapi.yml post /tags/upsert Insert or update a tag definition. ## Request Body | Field | Type | Description | |------------|-----------------|--------------------------------------------------| | `tag_data` | `Tag` | Tag data structure containing tag definition. | ## Response - **200**: Tag upserted successfully - Returns: `{ "tag_name": str, "tag": Tag, "available_values_added": List[str] }` - **400**: Bad Request (e.g., invalid tag data) - **500**: Internal Server Error ## Example ```json { "tag_data": { "name": "category", "description": "Document category classification", "type": "select", "available_values": ["invoice", "receipt", "contract"] } } ``` # Abort Progress Tracker Source: https://docs.deasylabs.com/api-reference/task-tracking/abort-progress-tracker /deasy-openapi.yml put /progress_tracker/abort/{tracker_id} Abort a progress tracker and update the tracker in Postgres. # Delete Progress Trackers Source: https://docs.deasylabs.com/api-reference/task-tracking/delete-progress-trackers /deasy-openapi.yml post /progress_tracker/delete Delete progress trackers from Postgres. Aborts any in-progress tasks before deleting. # Get Progress Trackers Source: https://docs.deasylabs.com/api-reference/task-tracking/get-progress-trackers /deasy-openapi.yml get /progress_tracker/get_progress_trackers Return progress trackers for the user. When tracker_ids is provided, only those trackers are returned. # Get Status Source: https://docs.deasylabs.com/api-reference/task-tracking/get-status /deasy-openapi.yml post /progress_tracker/task_status Get the status of a task by job ID. ## Request Body | Field | Type | Description | |----------|-------|--------------------------------------------------| | `job_id` | `str` | The unique identifier for the job to check. | ## Response - **200**: Task status retrieved successfully - Returns: `{ "percent_complete": float, "tags_created": Optional[int], "status": str }` - **404**: Job ID not found ## Example ```json { "job_id": "abc123-def456-ghi789" } ``` # Get Tracker Errors Source: https://docs.deasylabs.com/api-reference/task-tracking/get-tracker-errors /deasy-openapi.yml post /progress_tracker/errors Get the errors for a progress tracker. Returns both the legacy ``errors`` dict and the new ``typed_errors`` list, projected from a single DB fetch. The dict excludes job-scoped rows (``file_name IS NULL``) because ``classify_retry`` iterates dict keys as filenames; job-scoped errors remain visible via ``typed_errors``. # Delete Source: https://docs.deasylabs.com/api-reference/taxonomies/delete /deasy-openapi.yml delete /taxonomy/delete Delete a taxonomy (schema/hierarchy) by name. ## Query Parameters | Parameter | Type | Description | |-----------------|-------|-----------------------------------------| | `taxonomy_name` | `str` | Name of the taxonomy to delete. | ## Response - **200**: Taxonomy deleted successfully - Returns: `{ "taxonomy_name": str }` - **500**: Internal Server Error ## Example ``` DELETE /taxonomy/delete?taxonomy_name=document-categories ``` # Duplicate Source: https://docs.deasylabs.com/api-reference/taxonomies/duplicate /deasy-openapi.yml post /taxonomy/duplicate Create a copy of a taxonomy owned by the caller. The copy gets a fresh `taxonomy_id`, regenerated graph node/edge ids, the caller as owner, and reset run history/stats. `kind`, description, tag membership, sharing/permissions and project associations are inherited from the source. ## Request Body | Field | Type | Description | |----------------------|-------|--------------------------------------| | `source_taxonomy_id` | `str` | Taxonomy being copied (keyed by id). | | `new_taxonomy_name` | `str` | Name for the copy. | ## Response - **200**: Returns the new `DeasyTaxonomy`. - **400**: Empty `new_taxonomy_name`. - **403**: Caller is not the owner of the source. - **404**: Source taxonomy not found or not accessible. - **409**: `new_taxonomy_name` collides with an existing taxonomy. # Generate from Context Source: https://docs.deasylabs.com/api-reference/taxonomies/generate-from-context /deasy-openapi.yml post /taxonomy/generate_from_context Generate a taxonomy structure from user-provided context using AI. This endpoint allows users to describe their data and use case, and the system will generate an appropriate taxonomy structure (hierarchical or flat) without requiring actual data files. ## Request Body | Field | Type | Description | |--------------------|------------------|----------------------------------------------------------------------| | `user_context` | `str` | Free-form description of the data and intended use | | `industry` | `Optional[str]` | Industry context (e.g., "Healthcare", "Finance", "Legal") | | `data_type` | `Optional[str]` | Type of data (e.g., "documents", "customer records") | | `use_case` | `Optional[str]` | Specific use case description | | `taxonomy_type` | `str` | Type of taxonomy: "hierarchical" or "flat". Default: "hierarchical" | | `max_depth` | `Optional[int]` | Maximum depth for hierarchical taxonomies. Default: 3 | | `llm_profile_name` | `Optional[str]` | LLM profile to use. Defaults to system default if not specified | ## Response - **200**: Taxonomy generated successfully - Returns: `{ "taxonomy_data": Dict, "taxonomy_description": str }` - **400**: Invalid request or LLM configuration error - **500**: Internal Server Error ## Example ```json { "user_context": "I have customer support tickets with priority levels, categories, and resolution status", "industry": "Technology", "data_type": "support tickets", "use_case": "categorize and analyze support requests", "taxonomy_type": "hierarchical", "max_depth": 3 } ``` # List Source: https://docs.deasylabs.com/api-reference/taxonomies/list /deasy-openapi.yml post /taxonomy/list List all taxonomies (schemas/hierarchies) for the authenticated user. ## Request Body | Field | Type | Description | |----------------|-----------------------|----------------------------------------------------------------------| | `taxonomy_ids` | `Optional[List[str]]` | List of specific taxonomy IDs to retrieve. If omitted, returns all taxonomies. | ## Response - **200**: Taxonomies retrieved successfully - Returns: `{ "taxonomies": List[DeasyTaxonomy] }` - **500**: Internal Server Error ## Example ```json { "taxonomy_ids": ["document-categories", "sentiment-types"] } ``` # Suggest Source: https://docs.deasylabs.com/api-reference/taxonomies/suggest /deasy-openapi.yml post /suggest_schema Suggest a tag taxonomy based on file content and existing metadata. ## Request Body | Field | Type | Description | |-------------------------|------------------------|------------------------------------------------------------------------------| | `data_connector_name` | `str` | Name of the data connector (vector database profile) to use. | | `file_names` | `Optional[List[str]]` | List of specific files to analyze for the hierarchy suggestion. | | `dataslice_id` | `Optional[str]` | ID of a dataslice to pull files from for the suggestion. | | `progress_tracking_id` | `Optional[str]` | Custom tracking ID for monitoring the suggestion progress. | | `taxonomy_name` | `Optional[str]` | Name for the suggested schema/taxonomy. | | `current_tree` | `Optional[Dict]` | Existing hierarchy tree to build upon. | | `condition` | `Optional[Condition]` | Filtering condition to select specific files for analysis. | | `node` | `Optional[GraphNode]` | Node location in the existing hierarchy tree to build upon. Default: `{}` | | `user_context` | `Optional[str]` | User-provided context to guide the suggestion process. | | `context_level` | `Optional[str]` | Level at which to analyze content: `file` or `chunk`. Default: `"file"` | | `max_height` | `Optional[int]` | Maximum depth of the generated hierarchy tree. Default: `2` | | `use_existing_tags` | `Optional[bool]` | Whether to incorporate existing tags in suggestions. Default: `false` | | `use_extracted_tags` | `Optional[bool]` | Whether to use previously extracted tags. Default: `false` | | `use_mix_llm_and_source`| `Optional[bool]` | Whether to mix LLM-generated and source-based tags. Default: `false` | ## Response - **200**: Schema suggestion generated successfully - Returns: `{ "status_code": int, "message": str, "suggestion": Dict, "suggested_tags": Optional[Dict], "node": Optional[GraphNode], "tag_not_found_rates": Optional[Dict] }` - **500**: Internal Server Error ## Example ```json { "data_connector_name": "my-connector", "file_names": ["document1.pdf", "document2.pdf"], "context_level": "file", "max_height": 3, "use_existing_tags": true, "user_context": "Suggest categories for financial documents" } ``` # Upsert Source: https://docs.deasylabs.com/api-reference/taxonomies/upsert /deasy-openapi.yml post /taxonomy/upsert Insert or update a taxonomy (schema/hierarchy) in the database. ## Request Body | Field | Type | Description | |------------------------|------------------|----------------------------------------------------------------------| | `taxonomy_name` | `str` | Name of the taxonomy to upsert. | | `new_taxonomy_name` | `Optional[str]` | New name for the taxonomy (when updating). | | `taxonomy_description` | `Optional[str]` | Description of the taxonomy. | | `taxonomy_data` | `Optional[Dict]` | The taxonomy/hierarchy data structure. | ## Response - **200**: Taxonomy upserted successfully - Returns: `{ "taxonomy_name": str }` - **500**: Internal Server Error ## Example ```json { "taxonomy_name": "document-categories", "taxonomy_description": "Main document classification taxonomy", "taxonomy_data": { "invoice": ["purchase_invoice", "sales_invoice"], "receipt": ["expense_receipt", "payment_receipt"] } } ``` # Retrieve Versions Source: https://docs.deasylabs.com/api-reference/versioning/retrieve-versions /deasy-openapi.yml post /versioning/retrieve_versions List the file versions detected for a data connector. Returns the connector's clustered files, paginated with `limit` and `offset`. # Run Versioning Source: https://docs.deasylabs.com/api-reference/versioning/run-versioning /deasy-openapi.yml post /versioning/run Start a versioning job for a data connector. Queues the job and returns its job ID; poll the progress tracker endpoints with that ID to follow it. # Delete Workflow Source: https://docs.deasylabs.com/api-reference/workflows/delete-workflow /deasy-openapi.yml post /workflows/delete Delete a workflow # Execute Workflow Source: https://docs.deasylabs.com/api-reference/workflows/execute-workflow /deasy-openapi.yml post /workflows/execute Execute a workflow # List Workflows Source: https://docs.deasylabs.com/api-reference/workflows/list-workflows /deasy-openapi.yml post /workflows/list List workflows `project_external_id` is required. * Only workflows created directly under that project are returned; workflows in its sub-projects are not. Projects nest, workflows do not, so there is no recursive listing. * Archived and proposed workflows are included: `status` is lifecycle metadata, not a filter. * A `limit` above 100 is clamped to 100, not rejected. * A project that does not exist, or that the caller cannot access, returns 404 rather than 403. # Upsert Workflow Source: https://docs.deasylabs.com/api-reference/workflows/upsert-workflow /deasy-openapi.yml post /workflows/upsert Upsert a workflow # AI Readiness Source: https://docs.deasylabs.com/concepts/ai-readiness What makes a document set ready for AI, and how the platform measures and enforces it AI readiness is the question every use case has to answer before documents reach an agent, a RAG pipeline, or a lakehouse: is this data trustworthy, current, unique, and relevant enough to build on? Deasy Labs turns that question into measurable, file-level signals and a curated slice that enforces them. ## The Dimensions | Dimension | Question | Signal | | :----------------------- | :----------------------------------------- | :------------------------------------------------------------------------------------------ | | **Metadata coverage** | Do we know what each document is? | The standard quality tags (`Title`, `Author`, `Document Type`, `Document Date`) are present | | **Freshness** | Is the content still current? | Share of files not tagged `expired` | | **Uniqueness** | Is each document the single, latest copy? | Duplicates and superseded versions tagged `redundant` | | **Relevance** | Does the document belong to this use case? | Positive selection on extracted tags | | **Sensitivity** (opt-in) | Is the content safe for this audience? | `PII` / `PCI` / `PHI` rule-based tags | The readiness score is a summary; the products are the file-level signals and the filtered slice. Unrun dimensions are excluded from the score, not counted as zero. ## The Data Quality Status Tag Quality verdicts share one flag: `Data Quality Status`. A document tagged with it is either `redundant` (a duplicate or superseded copy, written by the platform's duplicate detection), `expired` (out of date for the use case's freshness rule), or otherwise flagged as unfit, such as `irrelevant`. A document with no `Data Quality Status` tag has passed every quality check, which is why the AI-ready slice uses a single `not_exists` condition. Files are tagged and excluded, never deleted, so every decision is auditable and reversible. ## Readiness Is Per Use Case A slice is a use case. Readiness scopes to a slice and applies that use case's own rules: which quality dimensions matter, what the freshness cutoff is, which document types are relevant, and whether sensitivity is part of the gate. The same file can be ready for one slice and not another. ```mermaid theme={"dark"} flowchart LR DOCS[One document set] --> S1[Slice A
contracts assistant
strict freshness + PII gate] DOCS --> S2[Slice B
internal legal search
PII allowed] S1 --> R1[Ready for A] S2 --> R2[Ready for B] ``` ## Where It Runs * **In the app**, the Data Quality screen walks the flow: Summary, Uniqueness, Freshness, Metadata, and AI Ready Data, with review steps for duplicates and expiring files. Background jobs appear in the Command Center. * **Through the SDK**, the whole flow is triggerable programmatically: run the scan, write the signals, and produce the curated slice inside your own pipeline. [Prepare an AI-Ready Dataset](/cookbooks/data-quality) is the complete headless walkthrough, ordered so the expensive classification step only ever runs on documents that survive the free filters. ## Next Steps The full readiness flow through the SDK. The freshness dimension in depth. The opt-in sensitivity gate. The slice conditions that enforce readiness. # Data Connectors Source: https://docs.deasylabs.com/concepts/data-connectors Connect to your document repositories and storage systems Data Connectors establish secure connections between the platform and your document storage. Once connected, the platform can discover documents, read content for metadata extraction, and write enriched metadata back to the source. ## Supported Data Connectors Connect to AWS S3 buckets for scalable cloud storage. Connect to Azure Blob containers. Connect to GCS buckets. Integrate with Microsoft 365 document libraries. Connect to Microsoft OneDrive sites and folders. Connect to PostgreSQL databases with pgvector support. Vector database integration for semantic search. ### Configuration Details | Source Type | Description | Key Configuration | Ideal Use Case | | :----------------------- | :------------------------------------------ | :---------------------------------------------- | :----------------------------------------------------------------- | | **Amazon S3** | AWS cloud object storage | Bucket name, Access Key, Secret Key | Large-scale document archives, cloud-native workflows | | **Azure Blob Storage** | Azure cloud object storage | Account, Container, Credentials | Azure-native document archives | | **Google Cloud Storage** | GCP cloud object storage | Bucket, Service Account | GCP-native document archives | | **SharePoint** | Microsoft 365 document management | Client ID, Client Secret, Tenant ID, Site Name | Enterprise document libraries, Office 365 environments | | **OneDrive** | Microsoft OneDrive cloud storage | Client ID, Client Secret, Tenant ID, Site | Personal and team drives in Microsoft 365 | | **PostgreSQL** | Relational database with pgvector extension | Host URL, Database name, User credentials, Port | Structured + unstructured hybrid data, existing database workflows | | **Qdrant** | Purpose-built vector database for AI | API Key, Collection name, URL | Semantic search applications, RAG pipelines | See the [Integrations Overview](/integrations/overview) for the complete matrix of sources, destinations, and supported file types. ## Key Features Create and manage multiple Data Connector connections Validate credentials before saving Switch between Data Connectors with one click Customize field mappings (filename key, text key, tags key) ## How Data Connectors Work ```mermaid theme={"dark"} flowchart LR subgraph sources [Your Data Sources] S3[Amazon S3] SP[SharePoint] PG[PostgreSQL] QD[Qdrant] end subgraph platform [Deasy Labs Platform] DC[Data Connector] PROC[Processing Engine] end S3 --> DC SP --> DC PG --> DC QD --> DC DC --> PROC ``` Select your storage type and provide the required credentials. Validate that the platform can access your documents before saving. Map your data fields (filename, text content, tags) to the platform's expected format. Your documents are now available for metadata extraction. *** ## Python SDK ```python theme={"dark"} from deasy import DeasyClient client = DeasyClient( base_url="https://unstructured.your-company.com/rest/unstructured", username="your-username", password="your-password", ) # Create an S3 connector connector = client.data_source.create( connector_name="my-s3-bucket", connector_body={ "type": "S3DataSourceManager", "name": "my-s3-bucket", "bucket_name": "my-documents", "aws_access_key_id": "YOUR_ACCESS_KEY", "aws_secret_access_key": "YOUR_SECRET_KEY", "region": "us-east-1", }, ) print(f"Created connector: {connector.profile_id}") ``` ```python theme={"dark"} # List all connectors (returned as a dict keyed by connector name) connectors = client.data_source.list() for name, config in connectors.connectors.items(): print(f"{name}: {config.type}") ``` ```python theme={"dark"} import uuid # Ingest documents from a connector (runs as a background job) job_id = str(uuid.uuid4()) client.data_source.ingest( data_connector_name="my-s3-bucket", file_names=["contracts/msa.pdf"], # Optional: omit to ingest everything job_id=job_id, ) # Track progress progress = client.task_status.get_status(job_id=job_id) print(f"Ingestion {progress.percent_complete:.0f}% complete ({progress.status})") ``` ```python theme={"dark"} # Delete a connector client.data_source.delete(connector_name="my-s3-bucket") print("Connector deleted") ``` *** ## API Reference Create a new data connector Update an existing connector configuration List all your data connectors Remove a data connector Ingest documents from a connector View ingested document metadata # Data Slices Source: https://docs.deasylabs.com/concepts/data-slices Filter documents for targeted processing and analysis Data Slices are filtered subsets of your data based on metadata. They allow you to focus on specific segments of your document repository without affecting the entire dataset. ## Benefits | Benefit | Description | | :---------------------- | :-------------------------------------------------------------------- | | **Targeted Processing** | Run extraction on only relevant documents (e.g., just 2024 contracts) | | **Focused Analysis** | View and analyze specific document categories | | **Efficient Workflows** | Avoid reprocessing already-enriched documents | | **Team Collaboration** | Share team-specific data slices with team members | | **Controllability** | Run downstream applications on controlled data | ## Data Slice Properties | Property | Description | | :----------------- | :--------------------------------------------------- | | **Name** | User-defined identifier | | **Description** | Optional notes about what's included | | **Data Connector** | Which Data Connector it's derived from | | **Document Count** | Number of files matching the conditions | | **Conditions** | Filter rules that define the slice based on metadata | ## How Data Slices Work ```mermaid theme={"dark"} flowchart LR subgraph source [Data Source] ALL[All Documents: 10,000] end subgraph slices [Data Slices, one per use case] S1[Support Chatbot KB: 1,247] S2[Sales Agent Brain: 892] S3[Compliance Review: 3,521] end ALL --> S1 ALL --> S2 ALL --> S3 ``` ## Example Data Slices A slice is a use case: it captures exactly the documents one AI application or initiative should see. | Slice Name | Feeds | Filter Conditions | | :------------------- | :----------------------- | :----------------------------------------------------------------------- | | "Support Chatbot KB" | Customer-facing chatbot | Document Type = Manual or FAQ AND no Data Quality Status flag AND no PII | | "Sales Agent Brain" | Account agent | Document Type = Contract or Pricing AND Data Quality Status not exists | | "Contracts 2024" | Legal analysis project | Document Type = Contract AND Year = 2024 | | "Compliance Review" | Human review queue | PII = true OR PHI = true | | "Needs Processing" | Remediation classify job | Required tags = Not found | The same document can appear in several slices and be excluded from others. The support chatbot never sees PII; the compliance queue sees nothing else. ## Creating Effective Data Slices Identify what subset of documents you need to work with. Select metadata fields and values that define your target documents. Use AND/OR logic to create precise filters. Check that the slice captures the expected number of documents. Use the slice in Projects or for targeted exports. ## Common Use Cases Filter to documents that haven't been processed yet Focus on documents containing sensitive information Analyze documents from specific time periods Examine all documents of a particular type *** ## Python SDK ```python theme={"dark"} from deasy import DeasyClient client = DeasyClient( base_url="https://unstructured.your-company.com/rest/unstructured", username="your-username", password="your-password", ) # Create a data slice with filter conditions slice_condition = { "condition": "AND", "children": [ {"tag": {"name": "document_type", "operator": "in", "values": ["Contract"]}}, {"tag": {"name": "year", "operator": "in", "values": ["2024"]}}, ], } slice = client.data_slice.create( data_connector_name="my-s3-bucket", dataslice_name="2024-contracts", description="All contracts from 2024", condition=slice_condition, ) print(f"Created slice: {slice.dataslice_id}") # Check how many files the slice captures count = client.data_slice.file_count( data_connector_name="my-s3-bucket", condition=slice_condition, ) ``` ```python theme={"dark"} # Documents missing required metadata (data-quality gate) needs_processing = client.data_slice.create( data_connector_name="my-s3-bucket", dataslice_name="needs-processing", condition={ "condition": "OR", "children": [ {"tag": {"name": tag, "operator": "in", "values": ["Not found"]}} for tag in ["document_type", "summary"] ], }, ) # Documents with detected PII sensitive = client.data_slice.create( data_connector_name="my-s3-bucket", dataslice_name="contains-pii", condition={ "tag": {"name": "ssn_detected", "operator": "in", "values": ["Yes"]}, }, ) ``` ```python theme={"dark"} # Export a data slice's metadata as CSV or JSON result = client.data_slice.export( data_connector_name="my-s3-bucket", dataslice_id="your-dataslice-id", export_format="csv", ) # Export the slice to a configured destination (e.g. a vector database) client.destination.export( destination_name="my-qdrant", dataslice_id="your-dataslice-id", export_level="chunk", export_nodes=True, ) ``` ```python theme={"dark"} # List all data slices slices = client.data_slice.list() for s in slices.dataslices: print(f"{s.name} ({s.id}): {s.data_points} docs") # Delete a data slice (by ID) client.data_slice.delete(dataslice_id="your-dataslice-id") print("Data slice deleted") ``` *** ## API Reference Create a new data slice List all your data slices Remove a data slice Export data from a slice # Destinations Source: https://docs.deasylabs.com/concepts/destinations Export enriched metadata to external systems Destinations are target systems where enriched metadata and document data can be exported. While Data Connectors bring documents into the platform, Destinations push enriched data out to vector databases, document management systems, and databases for downstream applications. ## Supported Destinations Relational database with vector support for hybrid search. Enrich original documents with metadata columns in Microsoft 365. Push enriched documents and metadata to OneDrive libraries. Write enriched output to S3 buckets. Write enriched output to GCS buckets. Serve curated chunks to RAG pipelines via slice export. ### Configuration Details | Destination Type | Description | Key Configuration | Ideal Use Case | | :----------------------- | :-------------------------------------- | :------------------------------------------------ | :------------------------------------------------- | | **PostgreSQL** | Relational database with vector support | Host URL, Port, Database, Collection, Credentials | Structured analytics, hybrid search | | **SharePoint** | Microsoft 365 document management | Client ID/Secret, Tenant ID, Site Name | Enriching original documents with metadata columns | | **OneDrive** | Microsoft OneDrive cloud storage | Client ID/Secret, Tenant ID, Site, Library folder | Team drives in Microsoft 365 | | **Amazon S3** | AWS cloud object storage | Bucket name, Access Key, Secret Key | Cloud-native archives and pipelines | | **Google Cloud Storage** | GCP cloud object storage | Bucket, Service Account | GCP-native archives and pipelines | | **Qdrant** | Vector database | URL, API Key, Collection | RAG serving collections | See the [Integrations Overview](/integrations/overview) for the complete matrix of sources, destinations, and supported file types. To export a data slice into a **vector database** (e.g. Qdrant) for RAG pipelines, use `client.data_slice.export_vdb(...)`. See [Data Slices](/concepts/data-slices). ## Export Options | Option | Description | Values | | :------------------ | :-------------------------------- | :------------------------------------------------------------------- | | **Export Level** | What data granularity to export | `file` (document-level), `chunk` (segment-level), `both` | | **Export Tags** | Specific metadata tags to include | List of tag names, or empty for all | | **Export Nodes** | Include vector embeddings | `true` / `false` | | **Export Metadata** | Include extracted metadata | `true` / `false` | | **Metadata Format** | How metadata is stored | `column_store` (separate columns), `json_store` (single JSON column) | **Export Processing:** * **Small Exports** (\< 100 files): Processed synchronously with immediate results * **Large Exports** (≥ 100 files): Processed in the background with progress tracking via `tracker_id` ## How Destinations Work ```mermaid theme={"dark"} flowchart LR subgraph platform [Deasy Labs Platform] META[Enriched Metadata] EXP[Export Engine] end subgraph destinations [Your Destinations] QD[Qdrant] PG[PostgreSQL] SP[SharePoint] end META --> EXP EXP --> QD EXP --> PG EXP --> SP ``` Configure the target system with the required credentials. Choose what data to export (file-level, chunk-level, specific tags). Decide between column store (separate columns) or JSON store (single JSON column). The platform sends enriched data to your destination system. *** ## Python SDK ```python theme={"dark"} from deasy import DeasyClient client = DeasyClient( base_url="https://unstructured.your-company.com/rest/unstructured", username="your-username", password="your-password", ) # Create a PostgreSQL destination destination = client.destination.create( connector_name="my-postgres-destination", connector_body={ "type": "PostgresNodeDestinationManager", "name": "my-postgres-destination", "url": "your-db-host.example.com", "port": 5432, "database_name": "analytics", "collection_name": "documents", "db_user": "postgres", "password": "YOUR_DB_PASSWORD", }, ) print(f"Created destination: {destination.profile_id}") ``` ```python theme={"dark"} # Export enriched data to a destination result = client.destination.export( destination_name="my-postgres-destination", data_connector_name="my-s3-bucket", export_level="chunk", # "file", "chunk", or "both" export_metadata=True, # Include extracted metadata metadata_format="json_store", ) print(f"Exported {result.success} files ({result.failed} failed)") # For large exports, track progress if result.tracker_id: status = client.task_status.get_status(job_id=result.tracker_id) print(f"Export {status.percent_complete:.0f}% complete ({status.status})") ``` ```python theme={"dark"} # List all destinations (returned as a dict keyed by destination name) destinations = client.destination.list() for name, config in destinations.connectors.items(): print(name) ``` ```python theme={"dark"} # Delete a destination client.destination.delete(connector_name="my-postgres-destination") print("Destination deleted") ``` *** ## API Reference Create a new destination List all your destinations Remove a destination Export enriched data to a destination # Metadata Source: https://docs.deasylabs.com/concepts/metadata The structured output from your document processing Metadata represents the actual extracted values that result from applying Tags to your documents. While Tags define *what* to extract, Metadata is the *extracted data* itself. ## Metadata Properties | Property | Description | Example | | :------------- | :------------------------------------- | :------------------------------------------------- | | **Values** | The extracted or classified value(s) | `["NDA", "Non-Disclosure Agreement"]` | | **Evidence** | Text snippet supporting the extraction | "This Non-Disclosure Agreement is entered into..." | | **Confidence** | AI confidence score (0-1) | `0.95` | The **Evidence** field shows exactly where the AI found the information, making it easy to verify extractions and understand the source. ## Metadata Levels | Level | Description | Use Case | | :-------------- | :------------------------------------------ | :--------------------------------------- | | **File-Level** | Aggregated metadata for the entire document | Document classification, search filters | | **Chunk-Level** | Granular metadata per text segment | Precise evidence location, RAG retrieval | ```mermaid theme={"dark"} flowchart TD subgraph doc [Document] FL[File-Level Metadata] subgraph chunks [Chunks] C1[Chunk 1 Metadata] C2[Chunk 2 Metadata] C3[Chunk 3 Metadata] end end FL --> C1 FL --> C2 FL --> C3 ``` ## Metadata Standardization The platform includes AI-powered standardization to clean and normalize extracted values: | Feature | Description | | :----------------------- | :----------------------------------------------------- | | **Deduplication** | Merge similar values (e.g., "Inc." and "Incorporated") | | **Normalization** | Standardize formats (dates, currencies, names) | | **Bulk Standardization** | Apply standardization across multiple tags | Standardization helps ensure consistency across your metadata, making it easier to search, filter, and analyze your documents. ## How Metadata Generation Works Documents are chunked and prepared for analysis. The AI applies your Tags to extract or classify information from each chunk. The system captures the text snippet that supports each extraction. Chunk-level metadata is aggregated to create file-level metadata. Optional normalization and deduplication cleans the results. ## Example Metadata Output For a contract document with a "Contract Type" classification tag: ```json theme={"dark"} { "tag": "Contract Type", "values": ["NDA"], "evidence": "This Non-Disclosure Agreement ('Agreement') is entered into as of January 1, 2024...", "confidence": 0.97 } ``` *** ## Python SDK ```python theme={"dark"} from deasy import DeasyClient client = DeasyClient( base_url="https://unstructured.your-company.com/rest/unstructured", username="your-username", password="your-password", ) # Generate metadata for specific documents result = client.metadata.generate.generate( data_connector_name="my-s3-bucket", file_names=["contract.pdf"], tag_names=["contract_type", "effective_date", "total_value"], ) print(result.message) ``` ```python theme={"dark"} import uuid # Generate metadata for all documents in a connector (background job) job_id = str(uuid.uuid4()) client.metadata.generate.generate_batch( data_connector_name="my-s3-bucket", tag_names=["contract_type", "effective_date", "total_value"], job_id=job_id, ) # Track progress progress = client.task_status.get_status(job_id=job_id) print(f"Classification {progress.percent_complete:.0f}% complete ({progress.status})") ``` ```python theme={"dark"} # List metadata for documents (dict keyed by file name) metadata = client.metadata.list( data_connector_name="my-s3-bucket", tag_names=["contract_type", "effective_date"], ) for file_name, tags in metadata.metadata.items(): print(f"{file_name}: {tags}") # Paginated listing for large datasets all_metadata, offset = {}, 0 while offset is not None: page = client.metadata.list_paginated( data_connector_name="my-s3-bucket", limit=200, offset=offset, ) all_metadata.update(page.metadata or {}) offset = page.next_offset print(f"Fetched metadata for {len(all_metadata)} files") ``` ```python theme={"dark"} # Manually upsert metadata (e.g. write a review status back) client.metadata.upsert( data_connector_name="my-s3-bucket", metadata={ "contract.pdf": { "contract_type": { "file_level": { "values": ["NDA"], "evidence": "Manually verified by legal team.", }, }, }, }, ) # Delete metadata for specific files client.metadata.delete( data_connector_name="my-s3-bucket", file_names=["old-contract.pdf"], ) ``` ```python theme={"dark"} import uuid # Preview AI-suggested standardizations for messy values suggestion = client.metadata.standardization_suggest( data_connector_name="my-s3-bucket", tag_names=["counterparty_name"], description="Consolidate company-name variants into one canonical legal name", processing_mode="entity_resolution", # or: deduplicate, smart_clustering, map_to_categories ) # Apply standardization across tags in bulk (background job) client.metadata.standardization_bulk( vdb_profile_name="my-s3-bucket", tag_names=["counterparty_name", "contract_type"], job_id=str(uuid.uuid4()), ) ``` *** ## API Reference Generate metadata for documents Generate metadata for multiple documents Create or update metadata List metadata for documents Paginated metadata listing Remove metadata # Projects Source: https://docs.deasylabs.com/concepts/projects Organize your work into focused workspaces Projects are organizational containers that group related work together, including Data Connectors, taxonomies, and sensitivity detection settings. Think of a Project as a workspace for a specific initiative. ## Example Use Cases A project is typically named after the AI application or initiative it feeds: * **Support Chatbot for Product Docs**, curating the current manuals and FAQs a customer-facing chatbot answers from, with PII detection enabled * **Sales Agent Brain**, equipping an account agent with signed contracts, pricing sheets, and renewal terms * **Contract Analysis 2024**, processing legal contracts for the annual review ## Project Components | Component | Description | Required | | :--------------------------- | :--------------------------------------- | :-------------------------- | | **Name** | Unique identifier for the project | Yes | | **Description** | Optional notes about the project purpose | No | | **Data Connectors** | One or more connected data repositories | At least one | | **Data Slice** | Optional filtered subset of data | No (defaults to "All Data") | | **Taxonomies** | One or more tag hierarchies to apply | No | | **Sensitive Data Detection** | Enable PII/PHI/PCI scanning | No | ## Project Structure ```mermaid theme={"dark"} flowchart TB subgraph project [Project: Contract Analysis 2024] DC[Data Connector: S3 Legal Bucket] DS[Data Slice: 2024 Contracts] TAX[Taxonomy: Contract Tags] SEN[Sensitivity Detection: PII Enabled] end DC --> DS DS --> TAX TAX --> SEN ``` ## Sensitivity Detection Options When creating a project, you can enable automatic sensitive data detection: | Type | Full Name | Examples | | :------ | :-------------------------------- | :------------------------------------------------------- | | **PII** | Personal Identifiable Information | Names, emails, addresses, phone numbers, SSN | | **PHI** | Protected Health Information | Medical records, diagnoses, prescriptions, insurance IDs | | **PCI** | Payment Card Industry Data | Credit card numbers, bank accounts, payment details | Enable sensitivity detection to automatically flag documents containing protected information. This is especially useful for compliance workflows in healthcare, finance, and legal industries. ## Workflow Example Give your project a descriptive name and optional description. Link one or more Data Connectors to bring in your documents. Optionally filter to specific documents using a Data Slice. Select which tag hierarchies to apply for metadata extraction. Turn on PII, PHI, or PCI scanning if needed for compliance. *** ## Python SDK ```python theme={"dark"} from deasy import DeasyClient client = DeasyClient( base_url="https://unstructured.your-company.com/rest/unstructured", username="your-username", password="your-password", ) # Create a project grouping connectors, taxonomies, and detection settings project = client.projects.create( name="Contract Analysis 2024", description="Processing legal contracts for the 2024 review", data_source=["my-s3-bucket"], taxonomy=["contract-analysis"], sensitive_data=True, # Enable PII/PHI/PCI scanning ) ``` ```python theme={"dark"} # List all projects projects = client.projects.list() # Get a single project project = client.projects.get("your-project-id") # Review detected sensitive data for a project sensitive = client.projects.get_sensitive_data("your-project-id") ``` ```python theme={"dark"} # Update a project client.projects.update( "your-project-id", description="Updated scope: includes Q1 2025 contracts", ) # Delete a project client.projects.delete("your-project-id") ``` *** ## API Reference Create a new project List all your projects Retrieve a single project Review detected sensitive data # Taxonomies & Tags Source: https://docs.deasylabs.com/concepts/taxonomies-tags Define what metadata to extract from your documents **Tags** are the metadata attributes you want to extract or classify from your documents. **Taxonomies** organize tags into hierarchical structures that define parent-child relationships. ## What is a Tag? A Tag defines a specific piece of information you want to capture from documents: | Property | Description | Example | | :------------------- | :----------------------------------------- | :----------------------------------------------------------- | | **Name** | The tag identifier | `Contract Type` | | **Description** | Instructions for the AI on what to extract | "Identify the type of legal agreement (NDA, MSA, SOW, etc.)" | | **Output Type** | How values are returned | String, Binary, Number, Date | | **Max Values** | How many values get returned | 1 to however many relevant values an AI can find | | **Available Values** | Predefined options (for classification) | `["NDA", "MSA", "SOW", "Employment Agreement"]` | | **Strategy** | Extraction method | LLM (AI), Regex (Pattern), Rule-based | ## Tag Types | Tag Type | How It Works | When to Use | Example | | :---------------------- | :-------------------------------------------------- | :-------------------------------------- | :----------------------------------------------------------------- | | **Classification Tags** | AI chooses from predefined list of values | When you have a known set of categories | Document Type: Contract, Invoice, Report | | **Extraction Tags** | AI extracts open-ended values from text | When the value is unpredictable | Contract Value: \$1,500,000 | | **Pattern Tags** | Regex plus context keywords that must appear nearby | For compliance, keyword-search, etc. | SSN: XXX-XX-XXXX, Email: [user@domain.com](mailto:user@domain.com) | **Classification** is best when you have a known set of categories. **Extraction** is best for unpredictable values like names, dates, or amounts. **Pattern** is best for structured data like phone numbers or SSNs. ## Three Ways to Define a Tag Every tag resolves its value through one of three strategies: | Strategy | How the value is decided | Cost | Example | | :---------------------------- | :-------------------------------------------------------- | :-------------------- | :--------------------------------------------- | | **AI (LLM)** | The model reads the document and follows your description | LLM call per document | `summary`, `contract_type` | | **Pattern (Regex + context)** | Deterministic matching, corroborated by context keywords | No LLM | `ssn_matches`, employee IDs | | **Rule-based** | Deterministic conditions over *other tags'* values | No LLM | `AI Agent Access = Restricted` if `PII = true` | Pattern tags are not bare regex. Each pattern carries **context keywords** (`context_items`) and a `required_context_matches` threshold: a nine-digit number only counts as an SSN when words like "social security" or "SSN" appear near it. That corroboration is what keeps precision high on real documents, and the platform can suggest both the pattern and its context keywords from your data. See [Precision Patterns for Sensitive Data](/cookbooks/precision-patterns). Rule-based tags evaluate after their input tags exist: if a rule matches, the value is resolved deterministically; if no rule matches, the tag falls back to its other configured strategy, pattern matching or LLM classification, depending on the tag definition. Because rules run at zero LLM cost, they are the right tool for policy verdicts derived from other signals (access gates, retention classes, readiness flags). ## Taxonomy Structure A Taxonomy enables hierarchical organization and conditional extraction. Child tags only get generated when parent conditions are met: ``` Document Type (Classification: Contract | Invoice | Report) ├── Contract │ ├── Contract Value (Extraction) │ ├── Parties Involved (Extraction) │ ├── Effective Date (Extraction) │ └── Termination Date (Extraction) ├── Invoice │ ├── Invoice Amount (Extraction) │ ├── Due Date (Extraction) │ └── Vendor Name (Extraction) └── Report ├── Report Category (Classification: Financial | Operational | Compliance) └── Report Period (Extraction) ``` ```mermaid theme={"dark"} flowchart TD DT[Document Type] --> C[Contract] DT --> I[Invoice] DT --> R[Report] C --> CV[Contract Value] C --> PI[Parties Involved] C --> ED[Effective Date] I --> IA[Invoice Amount] I --> DD[Due Date] I --> VN[Vendor Name] R --> RC[Report Category] R --> RP[Report Period] ``` In this taxonomy, the AI first classifies the document type, then only extracts the relevant child tags. A Contract won't have "Invoice Amount" extracted, which saves time and cost. ## Creating Effective Tags Give the AI specific instructions about what to extract. The better your description, the more accurate the extraction. Use Classification for known categories, Extraction for open-ended values, and Pattern for structured formats. For Classification tags, provide a complete list of possible values to improve accuracy. Group related tags hierarchically to enable conditional extraction and reduce unnecessary processing. *** ## Python SDK ```python theme={"dark"} from deasy import DeasyClient client = DeasyClient( base_url="https://unstructured.your-company.com/rest/unstructured", username="your-username", password="your-password", ) # Create or update tags, the metadata attributes you want to extract for tag in [ { "name": "contract_type", "description": "Type of contract (NDA, MSA, SLA, etc.)", "output_type": "string", "available_values": ["NDA", "MSA", "SLA", "Employment Agreement"], }, { "name": "parties", "description": "Names of all parties involved", "output_type": "string", "max_values": 10, }, { "name": "effective_date", "description": "When the contract becomes effective", "output_type": "date", }, { "name": "total_value", "description": "Total monetary value in USD", "output_type": "number", }, ]: result = client.tags.upsert(tag_data=tag) ``` ```python theme={"dark"} # Organize tags into a taxonomy, a node/edge graph where child tags # are only extracted when the parent condition is met taxonomy = client.taxonomy.upsert( taxonomy_name="contract-analysis", taxonomy_description="Extract key data from legal contracts", taxonomy_data={ "nodes": [ {"node_id": "1", "name": "contract_type", "active_values": ["NDA", "MSA", "SLA"]}, {"node_id": "2", "name": "total_value"}, ], "edges": [ # Only extract total_value for MSA contracts {"target_node_ids": ["2"], "conditions": [{"node_id": "1", "tag_value_name": "MSA"}]}, ], }, ) ``` ```python theme={"dark"} # Pattern tag: deterministic regex matching, no LLM cost client.tags.upsert(tag_data={ "name": "ssn_matches", "description": "US Social Security Numbers found in the document", "output_type": "string", "patterns": [{"pattern": r"\b\d{3}-\d{2}-\d{4}\b"}], }) # Rule-based tag: a deterministic verdict derived from other tags. # If a rule matches, the value resolves with no LLM call; if none # match, the tag falls back to its other configured strategy # (pattern matching or LLM classification, per the tag definition). client.tags.upsert(tag_data={ "name": "AI Agent Access", "description": "Whether AI agents may access this document", "output_type": "string", "available_values": ["Restricted", "Allowed"], "visual_rules": [ { "condition": { "condition": "OR", "children": [ {"tag": {"name": "PII (Personally Identifiable Information)", "operator": "in", "values": ["true"]}}, {"tag": {"name": "PHI (Protected Health Information)", "operator": "in", "values": ["true"]}}, ], }, "tag_value": "Restricted", }, ], }) ``` ```python theme={"dark"} # Get an AI-suggested taxonomy based on your actual documents suggestion = client.taxonomy.suggest( data_connector_name="my-s3-bucket", user_context="Medical patient records with diagnoses and prescriptions", ) print(suggestion.suggestion) # Get an AI-suggested regex pattern for a Pattern tag pattern = client.tags.pattern.suggest_patterns( pattern_description="US Social Security Number (XXX-XX-XXXX)", tag_data={"name": "ssn", "description": "US Social Security Number"}, ) print(f"Suggested pattern: {pattern.regex} (confidence: {pattern.confidence_score})") ``` ```python theme={"dark"} # List all taxonomies taxonomies = client.taxonomy.list() for t in taxonomies.taxonomies: print(t.taxonomy_name) # List all tags tags = client.tags.list() for tag in tags.tags: print(f"{tag.name} ({tag.output_type})") # Delete a taxonomy client.taxonomy.delete(taxonomy_name="old-taxonomy") # Delete a tag client.tags.delete(tag_name="unused-tag") ``` *** ## API Reference Create or update a tag List all your tags Remove a tag AI-powered pattern suggestions Create or update a taxonomy List all your taxonomies Remove a taxonomy AI-powered taxonomy suggestions # Workflows Source: https://docs.deasylabs.com/concepts/workflows Schedule and automate ingestion, classification, and export pipelines Workflows chain platform operations into scheduled, repeatable pipelines. Instead of manually re-running ingestion and classification as new documents arrive, define the sequence once and let it run on a cron cadence, or trigger it on demand. ## Anatomy of a Workflow | Field | Description | Example | | :---------- | :--------------------------------------------------------- | :-------------------------------- | | **Name** | Identifier shown in the UI's workflow table | `"Nightly prepare: contracts"` | | **Cadence** | Cron expression for the schedule | `"0 0 * * *"` (daily at midnight) | | **Stages** | Ordered list, each stage runs after the previous completes | Ingest, then Classify | | **Jobs** | API calls within a stage (endpoint + request body) | `/ocr/ingest`, `/classify_bulk` | ```mermaid theme={"dark"} flowchart LR CRON[Cron Trigger
0 0 * * *] --> S1 subgraph workflow [Workflow] S1[Stage 1: Ingest
/ocr/ingest] --> S2[Stage 2: Classify
/classify_bulk] end S2 --> META[Fresh Metadata] ``` Stages run **sequentially**; jobs inside a stage run together. Put ingestion and classification in separate stages so classification always sees the newly ingested documents. *** ## Python SDK ```python theme={"dark"} from deasy import DeasyClient client = DeasyClient( base_url="https://unstructured.your-company.com/rest/unstructured", username="your-username", password="your-password", ) # Nightly ingest + classify pipeline (same shape the web UI creates) result = client.workflows.upsert( workflow={ "name": "Nightly prepare: contracts", "description": "Ingest new documents and re-classify every night", "cadence": "0 0 * * *", # cron expression "stages": [ { "jobs": [ { "endpoint": "/ocr/ingest", "endpoint_request_body": { "data_connector_name": "my-s3-bucket", }, }, ], }, { "jobs": [ { "endpoint": "/classify_bulk", "endpoint_request_body": { "data_connector_name": "my-s3-bucket", }, }, ], }, ], }, ) ``` ```python theme={"dark"} # Execute a workflow immediately instead of waiting for the cadence client.workflows.execute(workflow_id="your-workflow-id") ``` ```python theme={"dark"} # List workflows workflows = client.workflows.list() # Delete a workflow client.workflows.delete(workflow_id="your-workflow-id") ``` *** ## API Reference Create or update a workflow Trigger a workflow on demand List all your workflows Remove a workflow # Bootstrap a Taxonomy with AI Source: https://docs.deasylabs.com/cookbooks/custom-taxonomy Use AI to automatically generate domain-specific taxonomies for your documents This cookbook shows you how to use the platform's AI to automatically generate custom taxonomies. Instead of manually defining tags one by one, you can describe your use case in natural language, and the AI will build a complete taxonomy for you. ## AI-Generated Taxonomies The `suggest` feature allows you to bootstrap complex taxonomies in seconds: ```python theme={"dark"} from deasy import DeasyClient client = DeasyClient( base_url="https://unstructured.your-company.com/rest/unstructured", username="your-username", password="your-password", ) # Describe what you want to extract. The AI grounds its suggestions # in the actual documents behind the data connector. suggestions = client.taxonomy.suggest( data_connector_name="my-documents", user_context=""" I need to analyze legal contracts. Extract key terms, financial obligations, risk indicators, important dates, and anything relevant for compliance. Focus on NDAs, MSAs, and employment agreements. """, taxonomy_name="legal-contracts", ) # Review the suggestions print("🤖 AI-Suggested Taxonomy:") print(suggestions.suggestion) ``` **Example output:** ``` 🤖 AI-Suggested Taxonomy: { "contract_type": { "description": "Primary contract classification: NDA, MSA, SLA, SOW, Employment, Lease", "type": "string" }, "parties": { "description": "All parties to the contract with their legal names", "type": "string" }, "effective_date": { "description": "Date when the contract becomes legally binding", "type": "date" }, "total_value": { "description": "Total monetary value of the contract in USD", "type": "number" } } ``` ## Save the Taxonomy Pass `auto_save=True` to store the suggested taxonomy directly, or review it first and create the tags yourself: ```python theme={"dark"} # Option A: save the suggestion in one call client.taxonomy.suggest( data_connector_name="my-documents", user_context="Analyze legal contracts for compliance", taxonomy_name="legal-contracts", auto_save=True, ) # Option B: review, then create tags from the suggestion for name, details in suggestions.suggestion.items(): client.tags.upsert(tag_data={ "name": name, "description": details["description"], "output_type": details["type"], }) ``` ## Refine with Sample Documents For higher accuracy, point to specific files in your data connector. The AI will analyze these documents to suggest relevant tags: ```python theme={"dark"} # The AI analyzes your files to suggest the most relevant tags suggestions = client.taxonomy.suggest( data_connector_name="my-s3-bucket", user_context="Extract key data from these vendor invoices", file_names=[ "invoices/sample-invoice-1.pdf", "invoices/sample-invoice-2.pdf" ] ) ``` ## Customize the Results You can modify the AI suggestions before creating the tags: ```python theme={"dark"} # Get suggestions response = client.taxonomy.suggest( data_connector_name="my-docs", user_context="Legal contract analysis", ) # Create tags from the suggestion for name, details in response.suggestion.items(): client.tags.upsert(tag_data={ "name": name, "description": details["description"], "output_type": details["type"], }) # Add a custom tag the AI might have missed client.tags.upsert(tag_data={ "name": "reviewed_by_legal", "description": "Whether this contract has been reviewed", "output_type": "binary", }) ``` ## Common Use Cases ```python theme={"dark"} client.taxonomy.suggest( user_context=""" Analyze quarterly financial reports. Extract revenue, earnings, growth metrics, risk factors, and forward guidance. """, data_connector_name="finance-docs" ) ``` ```python theme={"dark"} client.taxonomy.suggest( user_context=""" Process resumes for technical recruiting. Extract skills, experience level, education, and certifications. """, data_connector_name="hr-docs" ) ``` ```python theme={"dark"} client.taxonomy.suggest( user_context=""" Analyze insurance claims. Extract claim details, incident info, injury status, and potential fraud indicators. """, data_connector_name="claims-docs" ) ``` ## Next Steps Enrich libraries at the source with your taxonomy. Curate a serving collection with your taxonomy. Add sensitive data detection. Learn the fundamentals of taxonomies. # Prepare an AI-Ready Dataset Source: https://docs.deasylabs.com/cookbooks/data-quality Run the readiness flow headlessly and in cost order: expire stale files, drop duplicate versions, classify only the survivors, and slice to the relevant set The heart of Deasy Labs is turning a messy source into a curated, AI-ready dataset. In the app this is the Data Quality screen: Summary, Uniqueness, Freshness, Metadata, and AI Ready Data. The same flow runs headlessly through the SDK, so you can trigger a scan, write the signals, and produce a curated slice as part of your own pipeline. The steps run in cost order. Freshness and deduplication use metadata the platform captured for free at ingestion and content signals that need no LLM, so they shrink the corpus first. AI classification, the expensive step, runs only on the documents that survive. Only data that passes the gate reaches your destination. The platform does the heavy lifting. Ingestion captures source metadata (`Last Modified`, `File Type`, `Created By`, `Folder Structure`, and more) on every document, the versioning and duplicate detection engines work on content without LLM calls, and classification extracts the standard quality tags (`Title`, `Author`, `Document Type`, `Document Date`, `Version`). The SDK orchestrates and reuses that metadata; it never re-implements it. ## The Flow ```mermaid theme={"dark"} flowchart LR SCOPE[Scope
connector or slice] --> FRESH[Freshness rule
free source metadata] FRESH --> VERS[Versioning + dedup
content, no LLM] VERS --> CLS[Classify survivors
the only LLM step] CLS --> SLICE[AI-Ready Slice
relevant + unflagged] SLICE --> SHIP[Ship
SharePoint · Collibra · RAG] ``` ## Configuration The quality tags below are the platform's standard extracted tags. The coverage threshold and freshness cutoff are your use case's rules; the same file can be ready for one use case and not another. ```python theme={"dark"} from deasy import DeasyClient client = DeasyClient( base_url="https://unstructured.your-company.com/rest/unstructured", username="your-username", password="your-password", ) CONNECTOR = "my-sharepoint" QUALITY_TAGS = ["Title", "Author", "Document Type", "Document Date"] MIN_COVERAGE = 80 # percent of files that must carry each quality tag FRESHNESS_CUTOFF_YEAR = 2023 # documents older than this are expired for this use case MISSING_MARKERS = {"Not found", "", None} ``` ## Step 1. Expire Stale Files First `Last Modified` comes from the source system and was captured automatically at ingestion, so this filter costs nothing. Files older than your cutoff get `Data Quality Status = expired`, written through `metadata.upsert`, the same signal-write path the app's freshness service uses. Every file expired here is a file you never pay to classify. ```python theme={"dark"} def file_value(file_meta, tag): """Read a tag's first file-level value, tolerant of dict or model access.""" td = file_meta.get(tag) if isinstance(file_meta, dict) else getattr(file_meta, tag, None) if td is None: return None fl = td.get("file_level") if isinstance(td, dict) else getattr(td, "file_level", None) values = (fl.get("values") if isinstance(fl, dict) else getattr(fl, "values", None)) if fl else None if not values or values[0] in MISSING_MARKERS: return None return str(values[0]) metadata_by_file, offset = {}, 0 while offset is not None: page = client.metadata.list_paginated( data_connector_name=CONNECTOR, limit=200, offset=offset) metadata_by_file.update(page.metadata or {}) offset = page.next_offset total_files = len(metadata_by_file) expired = [] for file_name, file_meta in metadata_by_file.items(): modified = file_value(file_meta, "Last Modified") year = int(modified[:4]) if modified and modified[:4].isdigit() else None if year and year < FRESHNESS_CUTOFF_YEAR: expired.append(file_name) if expired: client.metadata.upsert( data_connector_name=CONNECTOR, metadata={ fn: {"Data Quality Status": {"file_level": { "values": ["expired"], "evidence": f"Last modified before the {FRESHNESS_CUTOFF_YEAR} cutoff for this use case.", }}} for fn in expired }, ) print(f"{len(expired)}/{total_files} file(s) expired before any LLM call") ``` `Last Modified` is the free first pass. After classification runs in Step 3, `Document Date` (the date a document states about itself) is the stronger signal; re-apply the cutoff with it for precision, as in [Keep Answers Current Over Time](/cookbooks/freshness-curation). ## Step 2. Drop Duplicates and Old Versions Versioning clusters documents that are versions of the same logical document by content, no LLM calls involved. Run it, then let the platform's selection keep the latest copy per group. ```python theme={"dark"} import time import uuid version_job = str(uuid.uuid4()) client.versioning.run(data_connector_name=CONNECTOR, job_id=version_job) while client.task_status.get_status(job_id=version_job).status == "in_progress": time.sleep(10) versions = client.versioning.retrieve_versions(data_connector_name=CONNECTOR) ``` Within each group, the latest version is determined from the strongest unambiguous signal: `Document Date` if present, then source-system timestamps like `Last Modified`, then extracted `Version` identifiers. The platform's duplicate detection (the Uniqueness tab in the app) applies that priority, recommends the canonical copy, and tags the rest `Data Quality Status = redundant`. Superseded copies are tagged and excluded, not deleted, so every decision is auditable and reversible. Exact duplicates (`...copy 2`, re-uploads) are caught by content hash the same way and also receive `Data Quality Status = redundant`. Every duplicate flagged here is another file you never pay to classify. Identity-based grouping gets even better once tags exist, so heavily duplicated sources are worth a second dedup pass after Step 3. ## Step 3. Classify Only the Survivors Now the expensive step, scoped to the smallest possible set. Create a working slice that excludes everything flagged so far, check tag coverage on it, and run the classification job against that slice only. Expired and redundant files never touch the LLM. ```python theme={"dark"} survivors = client.data_slice.create( data_connector_name=CONNECTOR, dataslice_name="needs-tagging", description="Unflagged files only. Scopes classification to the surviving set.", condition={ "tag": {"name": "Data Quality Status", "operator": "not_exists"}, }, ) # Coverage check on the survivors surviving_meta = { fn: fm for fn, fm in metadata_by_file.items() if not file_value(fm, "Data Quality Status") } coverage = {} for tag in QUALITY_TAGS: have = sum(1 for fm in surviving_meta.values() if file_value(fm, tag)) coverage[tag] = 100 * have / len(surviving_meta) if surviving_meta else 0 gaps = [t for t in QUALITY_TAGS if coverage[t] < MIN_COVERAGE] if gaps: job_id = str(uuid.uuid4()) client.metadata.generate.generate_batch( data_connector_name=CONNECTOR, dataslice_id=survivors.dataslice_id, # only the surviving files tag_names=gaps, job_id=job_id, ) while True: progress = client.task_status.get_status(job_id=job_id) if progress.status in ("completed", "failed", "aborted"): break print(f" Extracting missing metadata {progress.percent_complete:.0f}%...") time.sleep(10) print(f"Classification {progress.status}") ``` This mirrors what the Data Quality screen does when coverage is below threshold: it kicks off a background classification job, tracked in the Command Center. ## Step 4. Select the Relevant Files and Create the AI-Ready Slice Readiness has two sides. Quality says a file is trustworthy; relevance says it belongs to this use case. Both are decided by metadata: * **Relevance is a positive selection** on the tags you extracted. A contracts assistant wants `Document Type` in contracts and agreements, not canteen menus that happen to live in the same library. * **Quality is an exclusion** on `Data Quality Status`, the platform's shared quality flag. A document tagged with it is either `redundant` (a duplicate or superseded copy, written by the platform's duplicate detection), `expired` (out of date for this use case, written by the freshness rule), or otherwise flagged as unfit, such as `irrelevant`. A document with no `Data Quality Status` tag has passed every quality check. The slice combines both, using the same `not_exists` exclusion the platform's own AI-ready slice creation uses. Filtering to a slice is the heart of the experience: a slice is a use case. ```python theme={"dark"} ai_ready = client.data_slice.create( data_connector_name=CONNECTOR, dataslice_name="ai-ready-knowledge-base", description="Relevant document types only, no expired, redundant, or flagged files.", condition={ "condition": "AND", "children": [ # Relevance: only the document types this use case needs {"tag": {"name": "Document Type", "operator": "in", "values": ["Contract", "Agreement", "Policy"]}}, # Quality: no file carrying any Data Quality Status flag {"tag": {"name": "Data Quality Status", "operator": "not_exists"}}, ], }, ) print(f"AI-ready slice: {ai_ready.dataslice_id}") count = client.data_slice.file_count( data_connector_name=CONNECTOR, dataslice_id=ai_ready.dataslice_id, ) ``` ## Step 5. Ship It Only data that passes the gate reaches the destination. ```python theme={"dark"} # To a destination (SharePoint, SQL, S3) with metadata alongside client.destination.export( destination_name="my-destination", data_connector_name=CONNECTOR, dataslice_id=ai_ready.dataslice_id, export_level="file", export_metadata=True, ) # Or into a vector database for RAG client.data_slice.export_vdb( target_data_connector_name="rag-vectors", ori_data_connector_name=CONNECTOR, dataslice_id=ai_ready.dataslice_id, export_level="chunk", ) ``` ## How to Use This * **In your own pipeline.** Everything above is plain SDK calls: trigger the scan, write the signals, produce the slice, ship it. Schedule it with a [Workflow](/concepts/workflows) or your own orchestrator. * **Cost order matters.** Freshness and dedup are metadata and content checks; classification is the LLM spend. Filtering first means you tag the smallest possible set, and re-runs only classify what's new. * **Per use case.** Run it once per use case with different quality tags, thresholds, cutoffs, and relevance selections. The same file can be ready for one slice and not another. * **In the app.** The Data Quality screen runs this same flow interactively, with review steps for duplicates and expiring files, and the Command Center tracks the background jobs. ## Next Steps The full freshness flow, including the Document Date refinement. Add the opt-in sensitivity dimension to the gate. Ship the curated slice into a vector database. The condition syntax behind the gate. # Keep Answers Current Over Time Source: https://docs.deasylabs.com/cookbooks/freshness-curation Apply your freshness rule to platform-extracted dates, let the platform group versions, and keep agents answering from current documents An agent should answer from current documents, not a draft from 2003. Freshness in Deasy Labs is a use-case rule applied to metadata the platform already holds: you set the cutoff, the flow tags out-of-date files with `Data Quality Status = expired`, and the AI-ready slice excludes them. In the app this is the Freshness tab of the Data Quality screen; this cookbook runs the same flow headlessly through the SDK. You are not extracting dates yourself. Ingestion captures `Last Modified` from the source system automatically, and classification extracts `Document Date` (the date a document states about itself) as one of the platform's standard quality tags. This cookbook reuses those signals; expired files are tagged and excluded, not deleted, so every decision is auditable and reversible. ## The Flow ```mermaid theme={"dark"} flowchart LR TAGS[Platform date tags
Document Date · Last Modified] --> RULE[Your freshness rule
cutoff per use case] VERS[Platform versioning
version groups] --> RULE RULE --> EXP[Data Quality Status
expired] EXP --> SLICE[Authoritative slice
current documents only] ``` ## Configuration The cutoff is your use case's rule. A policy library might expire documents after three years; a contracts workspace might never expire signed agreements. ```python theme={"dark"} from deasy import DeasyClient client = DeasyClient( base_url="https://unstructured.your-company.com/rest/unstructured", username="your-username", password="your-password", ) CONNECTOR = "my-sharepoint" CUTOFF_YEAR = 2023 # documents older than this are expired for this use case ``` ## Step 1. Read the Platform's Date Tags `Document Date` is the strongest signal (what the document says about itself); `Last Modified` from the source system is the fallback. This is the same signal priority the platform's own latest-version selection uses. ```python theme={"dark"} def file_value(file_meta, tag): """Read a tag's first file-level value, tolerant of dict or model access.""" td = file_meta.get(tag) if isinstance(file_meta, dict) else getattr(file_meta, tag, None) if td is None: return None fl = td.get("file_level") if isinstance(td, dict) else getattr(td, "file_level", None) values = (fl.get("values") if isinstance(fl, dict) else getattr(fl, "values", None)) if fl else None if not values or values[0] in ("Not found", ""): return None return str(values[0]) meta, offset = {}, 0 while offset is not None: page = client.metadata.list_paginated( data_connector_name=CONNECTOR, tag_names=["Document Date", "Last Modified"], limit=200, offset=offset) meta.update(page.metadata or {}) offset = page.next_offset def best_year(file_meta): for tag in ("Document Date", "Last Modified"): value = file_value(file_meta, tag) if value and value[:4].isdigit(): return int(value[:4]) return None rows = [{"filename": fn, "year": best_year(fm)} for fn, fm in meta.items()] undated = [r for r in rows if r["year"] is None] print(f"{len(rows)} files, {len(undated)} without a usable date") ``` If many files lack a `Document Date`, run a classification job for it first, as in [Prepare an AI-Ready Dataset](/cookbooks/data-quality). Undated files are your curation backlog. ## Step 2. Group Versions with Platform Versioning The platform's versioning job clusters documents that are versions of the same logical document. It runs as a background job; the result is version groups you can combine with the date signals to find superseded copies. ```python theme={"dark"} import time import uuid job_id = str(uuid.uuid4()) client.versioning.run(data_connector_name=CONNECTOR, job_id=job_id) while True: progress = client.task_status.get_status(job_id=job_id) if progress.status in ("completed", "failed", "aborted"): break print(f" Version clustering {progress.percent_complete:.0f}%...") time.sleep(10) versions = client.versioning.retrieve_versions(data_connector_name=CONNECTOR) ``` Choosing the canonical copy within each group is what the platform's duplicate detection does on the Data Quality screen: it recommends the latest version by `Document Date`, then source timestamps, then extracted `Version`, and flags the rest. Review those recommendations in the app's Uniqueness tab. ## Step 3. Apply the Cutoff and Write the Signal Files older than your cutoff get `Data Quality Status = expired`, written through `metadata.upsert`. This is the same signal-write path the app's freshness service uses, so the result is indistinguishable from running the flow in the UI. ```python theme={"dark"} expired = [r["filename"] for r in rows if r["year"] and r["year"] < CUTOFF_YEAR] if expired: client.metadata.upsert( data_connector_name=CONNECTOR, metadata={ fn: {"Data Quality Status": {"file_level": { "values": ["expired"], "evidence": f"Dated before the {CUTOFF_YEAR} cutoff for this use case.", }}} for fn in expired }, ) print(f"Marked {len(expired)} file(s) expired") # Verify the write by reading the tag back check, offset = {}, 0 while offset is not None: page = client.metadata.list_paginated( data_connector_name=CONNECTOR, tag_names=["Data Quality Status"], limit=200, offset=offset) check.update(page.metadata or {}) offset = page.next_offset confirmed = sum( 1 for fn in expired if file_value(check.get(fn, {}), "Data Quality Status") == "expired") print(f"Verified {confirmed}/{len(expired)} expired tags persisted.") ``` ## Step 4. Slice to Current Documents Freshness is the share of files not tagged `expired`. The `not_exists` condition below excludes every file carrying any `Data Quality Status` value, whether `expired` from this rule, `redundant` from the platform's duplicate detection, or otherwise flagged. Only unflagged documents make the authoritative set, the same condition the platform's AI-ready slice creation uses. ```python theme={"dark"} authoritative = client.data_slice.create( data_connector_name=CONNECTOR, dataslice_name="authoritative-current", description="Current documents only. Safe for agent consumption.", condition={ "tag": {"name": "Data Quality Status", "operator": "not_exists"}, }, ) total = len(rows) print("=" * 50) print(f" FRESHNESS SCORECARD {CONNECTOR}") print("=" * 50) print(f" Total documents {total:>5}") print(f" Expired {len(expired):>5}") print(f" Undated {len(undated):>5} curation gap") print(f" Freshness {100 * (total - len(expired)) / total if total else 0:>5.1f}%") ``` ## How to Use This * **As an agent guardrail.** Point agents and RAG pipelines at the authoritative slice so they never answer from expired content. * **Per use case.** The cutoff belongs to the use case, not the corpus. The same file can be current for one slice and expired for another. * **On a schedule.** Re-run the flow with a [Workflow](/concepts/workflows) so freshness keeps up as documents change. ## Next Steps The full readiness flow this rule plugs into. Let users scope questions to current documents. Schedule the freshness re-scan. The condition syntax behind the authoritative slice. # Scope Chatbot Answers with Metadata Source: https://docs.deasylabs.com/cookbooks/metadata-filtered-rag Build a RAG chatbot where extracted tags become retrieval filters that scope every answer The metadata tags Deasy Labs extracts become retrieval filters for a RAG chatbot. A user can ask "answer using only Supplier Agreements" or "only documents governed by California law", and the retriever narrows the candidate pool before the LLM sees a single chunk. Filtered retrieval keeps answers grounded in the right documents instead of the most superficially similar ones. ## What You'll Build ```mermaid theme={"dark"} flowchart LR DL[Deasy Labs
chunks + tags] --> IDX[Vector Index
tags as payload] Q[User Question
+ optional filter] --> IDX IDX --> CTX[Filtered Context] CTX --> LLM[LLM Answer] ``` ## Prerequisites * A data connector with ingested and classified documents * Python 3.9+ with `qdrant-client`, `fastembed`, and `openai` installed ```bash theme={"dark"} pip install deasy_sdk-*.whl qdrant-client fastembed openai ``` ## Step 1. Pull Chunk Text and File-Level Tags Fetch each file's tags with `metadata.list_paginated`, then the chunk text with `data_source.list_ingested_data`. Each chunk record carries its file's tag values. ```python theme={"dark"} from deasy import DeasyClient client = DeasyClient( base_url="https://unstructured.your-company.com/rest/unstructured", username="your-username", password="your-password", ) CONNECTOR = "my-s3-bucket" FILTERABLE_TAGS = ["contract_type", "counterparty_name", "governing_law"] def file_level_value(file_meta, tag): """Read a tag's first file-level value, tolerant of dict or model access.""" td = file_meta.get(tag) if isinstance(file_meta, dict) else getattr(file_meta, tag, None) if td is None: return None fl = td.get("file_level") if isinstance(td, dict) else getattr(td, "file_level", None) values = (fl.get("values") if isinstance(fl, dict) else getattr(fl, "values", None)) if fl else None if not values or values == ["Not found"]: return None return str(values[0]) def chunk_text(chunk): """Extract chunk text, tolerant of varying chunk shapes.""" if not isinstance(chunk, dict): return "" for key in ("text", "content", "document_text", "page_content"): value = chunk.get(key) if isinstance(value, str) and value.strip(): return value.strip() return "" # Tags per file meta, offset = {}, 0 while offset is not None: page = client.metadata.list_paginated( data_connector_name=CONNECTOR, tag_names=FILTERABLE_TAGS, limit=200, offset=offset) meta.update(page.metadata or {}) offset = page.next_offset filenames = sorted(meta.keys()) # Chunk text per file resp = client.data_source.list_ingested_data( data_connector_name=CONNECTOR, file_names=filenames, group_by="file", limit=len(filenames)) ingested = resp.metadata or {} records = [] for fn in filenames: tag_values = {t: file_level_value(meta.get(fn, {}), t) for t in FILTERABLE_TAGS} chunks = ingested.get(fn) if not isinstance(chunks, dict): continue for chunk_id, chunk in chunks.items(): text = chunk_text(chunk) if text: records.append({"filename": fn, "text": text, **tag_values}) print(f"Collected {len(records)} chunks from {len(filenames)} files") ``` ## Step 2. Embed and Index with Tags as Payload Every chunk goes into the vector index with its tag values attached as payload fields. ```python theme={"dark"} from qdrant_client import QdrantClient from qdrant_client.models import Distance, VectorParams, PointStruct from fastembed import TextEmbedding embedder = TextEmbedding(model_name="BAAI/bge-small-en-v1.5") vectors = list(embedder.embed([r["text"] for r in records])) qdrant = QdrantClient(":memory:") # swap for your persistent cluster in production qdrant.create_collection( "documents", vectors_config=VectorParams(size=len(vectors[0]), distance=Distance.COSINE), ) qdrant.upsert("documents", points=[ PointStruct( id=i, vector=vectors[i].tolist(), payload={"filename": r["filename"], "text": r["text"], **{t: r[t] for t in FILTERABLE_TAGS}}, ) for i, r in enumerate(records) ]) print(f"Indexed {len(records)} chunks") ``` ## Step 3. Ask, with Optional Metadata Filters The filter narrows the candidate pool before similarity search runs. Without it, retrieval considers every chunk in the corpus. ```python theme={"dark"} import os from openai import OpenAI from qdrant_client.models import Filter, FieldCondition, MatchValue openai_client = OpenAI(api_key=os.environ["OPENAI_API_KEY"]) def ask(question, filters=None, top_k=5): query_vector = next(iter(embedder.embed([question]))).tolist() query_filter = None if filters: query_filter = Filter(must=[ FieldCondition(key=k, match=MatchValue(value=v)) for k, v in filters.items() ]) hits = qdrant.query_points( "documents", query=query_vector, limit=top_k, query_filter=query_filter).points if not hits: return "No matching context. The filter may be too narrow." context = "\n\n---\n\n".join( f"Source: {h.payload['filename']}\n{h.payload['text']}" for h in hits) response = openai_client.chat.completions.create( model="gpt-4.1", temperature=0, max_tokens=500, messages=[ {"role": "system", "content": "Answer only from the provided context. If the answer is absent, say so."}, {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}, ]) return response.choices[0].message.content # Unscoped: retrieval sees every document print(ask("What are the key obligations and termination terms?")) # Scoped: retrieval only sees Supplier Agreements print(ask( "What are the key obligations and termination terms?", filters={"contract_type": "Supplier Agreement"}, )) ``` ## Step 4. Compare Filtered and Unfiltered Retrieval Run the same question both ways and measure how the filter narrows the candidate pool. Pick a filter value that actually exists in your data. ```python theme={"dark"} contract_types = sorted({r["contract_type"] for r in records if r["contract_type"]}) demo_filter = {"contract_type": contract_types[0]} if contract_types else None question = "What are the key obligations and termination terms?" hits_open = qdrant.query_points( "documents", query=next(iter(embedder.embed([question]))).tolist(), limit=5).points hits_filtered = qdrant.query_points( "documents", query=next(iter(embedder.embed([question]))).tolist(), limit=5, query_filter=Filter(must=[ FieldCondition(key=k, match=MatchValue(value=v)) for k, v in (demo_filter or {}).items() ])).points if demo_filter else [] candidate_pool = sum( 1 for r in records if demo_filter and all(r.get(k) == v for k, v in demo_filter.items())) print("=" * 52) print(" CHATBOT / METADATA-FILTER SCORECARD") print("=" * 52) print(f" Indexed chunks : {len(records)}") print(f" Retrieved (no filter) : {len(hits_open)} chunks " f"across {len({h.payload['filename'] for h in hits_open})} files") if demo_filter: print(f" Candidate pool w/ filter : {candidate_pool} chunks (filter: {demo_filter})") print(f" Retrieved (filtered) : {len(hits_filtered)} chunks " f"across {len({h.payload['filename'] for h in hits_filtered})} files") print("=" * 52) ``` ## Why Filtered Retrieval Wins | | Unfiltered | Filtered by tag | | :------------- | :---------------------------------------------- | :-------------------------------------------------------------- | | Candidate pool | Every chunk in the corpus | Only chunks whose file matches the filter | | Failure mode | Superficially similar but wrong-document chunks | Empty result if the filter is too narrow | | Best for | Broad exploratory questions | Scoped questions ("in our NDAs...", "under California law\...") | Any tag you extract becomes a filter for free. The richer your taxonomy, the more precisely users can scope questions. ## How to Use This * **Scope answers by any extracted tag.** Pass `filters={"governing_law": "California"}` or any other tag/value pair. * **Swap the corpus.** Change `CONNECTOR` and `FILTERABLE_TAGS`; the pipeline adapts to whatever tags exist. * **Productionize.** Point at a persistent Qdrant cluster instead of `:memory:`, or skip the manual indexing entirely and use `data_slice.export_vdb` as in the [Clean Up a RAG Index](/cookbooks/qdrant-to-qdrant). If your tag names contain spaces, slugify them before using them as payload keys; some vector stores cannot filter on keys with spaces. ## Next Steps Use the platform's own vector export instead of indexing manually. Design the tags that power your filters. Keep the chatbot answering from current documents only. Feed the index only documents that pass quality checks. # Protect Sensitive Data Source: https://docs.deasylabs.com/cookbooks/pii-detection Run the platform's built-in sensitive data detection and gate what AI systems are allowed to see A customer-facing agent must not see PII. Deasy Labs ships a proprietary sensitive-data classification engine with 40+ built-in classifiers (Social Security Number, Credit Card, Date of Birth, Physical Address, and more) that combines pattern matching with AI context, understanding meaning, not just format. This cookbook runs that detection through the SDK and gates a data slice on the result, so restricted content never reaches downstream AI systems. Sensitivity is an opt-in component of readiness, chosen per use case. In the app you enable it when creating a [Project](/concepts/projects); here you run it headlessly. You do not write detection logic. The classifiers are built in, and when sensitivity detection runs, the platform automatically adds three rule-based tags: `PII (Personally Identifiable Information)`, `PCI (Payment Card Industry Data)`, and `PHI (Protected Health Information)`. PII detection currently supports English documents only. ## The Flow ```mermaid theme={"dark"} flowchart LR CAT[Built-in classifiers
40+ sensitivity tags] --> RUN[Classification job
sensitivity strategy] RUN --> ROLL[Rule-based tags
PII · PCI · PHI] ROLL --> GATE[Gated slice
no sensitive content] GATE --> AI[Agents · RAG · exports] ``` ## Step 1. Review the Built-In Sensitivity Catalog ```python theme={"dark"} from deasy import DeasyClient client = DeasyClient( base_url="https://unstructured.your-company.com/rest/unstructured", username="your-username", password="your-password", ) CONNECTOR = "my-sharepoint" # The platform's built-in sensitivity classifiers catalog = client.tags.list_sensitivity_defaults() ``` The catalog covers identity, financial, health, and contact data: Social Security Number, Credit Card, Tax ID, Date of Birth, Full Name, Physical Address, Phone Number, Policy Number, and more. ## Step 2. Run Sensitivity Detection Include the sensitivity tags in a classification job. The platform runs its sensitivity strategy at the highest merge priority and writes the detected values, plus the `PII`, `PCI`, and `PHI` rule-based tags, as file-level metadata. ```python theme={"dark"} import time import uuid SENSITIVITY_TAGS = ["Social Security Number", "Credit Card", "Date of Birth", "Full Name"] job_id = str(uuid.uuid4()) client.metadata.generate.generate_batch( data_connector_name=CONNECTOR, tag_names=SENSITIVITY_TAGS, job_id=job_id, ) while True: progress = client.task_status.get_status(job_id=job_id) if progress.status in ("completed", "failed", "aborted"): break print(f" Sensitivity scan {progress.percent_complete:.0f}%...") time.sleep(10) print(f"Scan {progress.status}") ``` ## Step 3. Review What Was Found Every detection carries evidence and confidence, so findings are auditable. ```python theme={"dark"} results = client.metadata.list( data_connector_name=CONNECTOR, tag_names=["PII (Personally Identifiable Information)"], ) flagged = [] for file_name, tags in (results.metadata or {}).items(): pii = tags.get("PII (Personally Identifiable Information)") values = pii.file_level.values if pii and pii.file_level else [] if values and str(values[0]).lower() in ("true", "yes"): flagged.append(file_name) print(f"{len(flagged)} file(s) contain PII") ``` ## Step 4. Encode the Policy as a Rule-Based Tag Turn the detection results into a deterministic access verdict. A rule-based tag evaluates conditions over other tags' values at zero LLM cost: if a rule matches, the value resolves deterministically, and if none match, the tag falls back to its other configured strategy, pattern matching or LLM classification, depending on the tag definition. ```python theme={"dark"} client.tags.upsert(tag_data={ "name": "AI Agent Access", "description": "Whether AI agents may access this document", "output_type": "string", "available_values": ["Restricted", "Allowed"], "visual_rules": [ { "condition": { "condition": "OR", "children": [ {"tag": {"name": "PII (Personally Identifiable Information)", "operator": "in", "values": ["true"]}}, {"tag": {"name": "PHI (Protected Health Information)", "operator": "in", "values": ["true"]}}, ], }, "tag_value": "Restricted", }, ], }) ``` The policy now lives on the documents themselves: re-running classification re-evaluates the rule, and every downstream system reads one tag instead of re-deriving the logic. ## Step 5. Gate the Slice Build the use case's slice so restricted content is excluded. This is the sensitivity component of the AI-ready gate: which dimensions matter, including whether sensitivity is part of the gate, is decided per use case. ```python theme={"dark"} safe_for_agents = client.data_slice.create( data_connector_name=CONNECTOR, dataslice_name="agent-safe-knowledge", description="No PII, PCI, or PHI. Approved for customer-facing agent access.", condition={ "condition": "AND", "children": [ {"tag": {"name": "PII (Personally Identifiable Information)", "operator": "not_exists"}}, {"tag": {"name": "PCI (Payment Card Industry Data)", "operator": "not_exists"}}, {"tag": {"name": "PHI (Protected Health Information)", "operator": "not_exists"}}, ], }, ) print(f"Gated slice: {safe_for_agents.dataslice_id}") ``` Export this slice, and only it, to the systems your agents read from. Documents with sensitive content stay behind the gate for review. ## Custom Patterns For organization-specific identifiers (employee IDs, claim numbers, internal codes), describe the pattern and let the platform engineer it, including the context keywords that keep precision high. The full loop, with generated test cases that prove the pattern before it runs at scale, is in [Precision Patterns for Sensitive Data](/cookbooks/precision-patterns). ```python theme={"dark"} suggestion = client.tags.pattern.suggest_patterns( pattern_description="Internal employee ID in the format EMP-XXXXXX", tag_data={"name": "employee_id", "description": "Internal employee identifier"}, ) print(f"Suggested pattern: {suggestion.regex} (confidence: {suggestion.confidence_score})") client.tags.upsert(tag_data={ "name": "employee_id", "description": "Internal employee identifier", "output_type": "string", "patterns": [{"pattern": suggestion.regex}], }) ``` ## PII Categories Reference | Category | Examples | Typical Risk | | :-------------- | :--------------------------------------- | :----------- | | **Identity** | SSN, Passport, Driver's License | Critical | | **Financial** | Credit Card, Bank Account, Tax ID | Critical | | **Health** | Medical Records, Insurance ID, Diagnoses | High | | **Contact** | Email, Phone, Address | Medium | | **Demographic** | Age, Gender, Religion | Low-Medium | ## How to Use This * **Per use case.** An internal legal workspace may allow PII that a customer-facing agent must never see. Gate each slice with its own rules. * **In a project.** Enable Sensitive Data Detection when creating a [Project](/concepts/projects) to make the scan part of the standard workspace setup. * **Composed with quality.** Combine the sensitivity conditions with the `Data Quality Status` exclusion from [Prepare an AI-Ready Dataset](/cookbooks/data-quality) for a gate that covers both. ## Next Steps Add the quality dimensions to the gate. Enable sensitivity detection per workspace. Ship the gated slice to a vector database. Tag strategies: LLM, Pattern, Rule-based, Sensitivity. # Precision Patterns for Sensitive Data Source: https://docs.deasylabs.com/cookbooks/precision-patterns Let the platform engineer regex plus context keywords from your data, then prove precision with generated test cases A bare regex is a false-positive machine: every nine-digit number becomes an SSN, every sixteen-digit number a credit card. Deasy Labs pattern tags avoid that with **context keywords**: the pattern only counts when corroborating words appear near the match. And you don't engineer either part by hand. Describe what matters, and the platform suggests the regex, the context keywords, and the corroboration threshold from your data, then generates test cases to prove precision before anything runs at scale. ## The Loop ```mermaid theme={"dark"} flowchart LR DESC[Describe
what to detect] --> SUG[Suggest
regex + context keywords] SUG --> TEST[Generate + evaluate
test cases] TEST -->|failures feed back| SUG TEST --> TAG[Create the tag
run at scale] ``` ## Step 1. Suggest the Pattern from Your Data Give the platform a description and real examples. It returns the regex, the context keywords, the corroboration threshold, and an explanation of its reasoning. ```python theme={"dark"} from deasy import DeasyClient client = DeasyClient( base_url="https://unstructured.your-company.com/rest/unstructured", username="your-username", password="your-password", ) suggestion = client.tags.pattern.suggest_patterns( pattern_description="US Social Security Numbers in employee records", tag_data={"name": "ssn", "description": "US Social Security Numbers"}, examples=["SSN: 123-45-6789", "Social Security Number 987-65-4321"], negative_examples=["Order #123-45-6789", "Part no. 987-65-4321"], ) print(f"Regex: {suggestion.regex}") print(f"Context keywords: {suggestion.context_items}") print(f"Required matches: {suggestion.required_context_matches}") print(f"Reasoning: {suggestion.explanation}") ``` The `negative_examples` are what teach it precision: the same digit shape in an order number must not match, so the suggestion leans on context keywords like "SSN" and "social security" to separate the two. ## Step 2. Prove It with Test Cases Generate test cases covering matches, near-misses, and edge cases, then evaluate the pattern against them before it touches production data. ```python theme={"dark"} generated = client.tags.pattern.generate_test_cases( pattern_descriptions=["US Social Security Numbers in employee records"], existing_patterns=[suggestion.regex], context_keywords=suggestion.context_items, tag_name="ssn", ) evaluation = client.tags.pattern.evaluate_test_cases( patterns=[{"pattern": suggestion.regex, "context_items": suggestion.context_items, "required_context_matches": suggestion.required_context_matches}], test_cases=[ {"text": tc.text, "should_match": tc.should_match, "category": tc.category} for tc in generated.test_cases ], ) failures = [tc for tc in evaluation.evaluated_test_cases if tc.actual_match != tc.should_match] print(f"{len(evaluation.evaluated_test_cases) - len(failures)}" f"/{len(evaluation.evaluated_test_cases)} test cases pass") for tc in failures: print(f" MISS [{tc.category}] {tc.text!r}") ``` If a case fails, feed it back: call `suggest_patterns` again with the failing text as `current_wrong_result` or an added negative example, and re-evaluate. The loop converges in a round or two. ## Step 3. Create the Tag and Run It The tag carries the full pattern configuration. Detection is deterministic and costs no LLM calls at classification time. ```python theme={"dark"} client.tags.upsert(tag_data={ "name": "ssn", "description": "US Social Security Numbers", "output_type": "string", "patterns": [{ "pattern": suggestion.regex, "context_items": suggestion.context_items, "required_context_matches": suggestion.required_context_matches, }], }) ``` Run it in a classification job like any other tag, then gate slices on the result as in [Protect Sensitive Data](/cookbooks/pii-detection). ## Why Context Keywords Matter | | Bare regex | Pattern + context keywords | | :------------------------------------ | :-------------------------------------- | :--------------------------------- | | `123-45-6789` in "SSN: 123-45-6789" | Match | Match | | `123-45-6789` in "Order #123-45-6789" | False positive | No match, no corroborating context | | Cost at scale | Reprocessing false positives downstream | Deterministic, precise, no LLM | ## Next Steps The built-in classifiers and the gated slice. The three tag strategies side by side. # Clean Up a RAG Index Source: https://docs.deasylabs.com/cookbooks/qdrant-to-qdrant Enrich an existing Qdrant collection with metadata and serve a curated AI-ready collection, kept fresh over time Your vector database already holds chunks; what it lacks is curation. This cookbook connects an existing Qdrant collection as a source, tags its content with AI-extracted metadata, and exports a curated slice into a clean serving collection. Retrieval moves from "everything, by similarity alone" to "gated documents, filterable by tags". A scheduled workflow keeps the serving collection maintained as the source evolves. ## What You'll Build ```mermaid theme={"dark"} flowchart LR Q1[Qdrant
raw collection] --> DL[Deasy Labs
classify + curate] DL -->|AI-ready slice| Q2[Qdrant
serving collection] Q2 --> RAG[RAG / Agents] CRON[Nightly workflow] -.->|maintain| DL ``` ## Prerequisites * A Qdrant instance with an existing collection of document chunks * Python 3.9+ ```bash theme={"dark"} pip install deasy_sdk-*.whl ``` ## Step 1. Connect Both Collections The source is your existing collection; the target is the curated serving collection your RAG pipeline will read from. `filename_key` and `text_key` tell the platform which payload fields hold the document name and chunk text. ```python theme={"dark"} from deasy import DeasyClient client = DeasyClient( base_url="https://unstructured.your-company.com/rest/unstructured", username="your-username", password="your-password", ) client.data_source.create( connector_name="raw-vectors", connector_body={ "type": "QdrantVectorDBManager", "name": "raw-vectors", "url": "https://your-cluster.qdrant.io", "api_key": "YOUR_QDRANT_API_KEY", "collection_name": "all_documents", "filename_key": "filename", "text_key": "text", }, ) client.data_source.create( connector_name="serving-vectors", connector_body={ "type": "QdrantVectorDBManager", "name": "serving-vectors", "url": "https://your-cluster.qdrant.io", "api_key": "YOUR_QDRANT_API_KEY", "collection_name": "ai_ready_documents", }, ) print("✓ Source and serving collections connected") ``` ## Step 2. Tag the Existing Content Classification runs over the chunks already in your collection. No re-ingestion of source files needed. ```python theme={"dark"} import time import uuid for tag in [ { "name": "document_type", "description": "Type of document (contract, policy, report, manual, etc.)", "output_type": "string", "available_values": ["contract", "policy", "report", "manual", "other"], }, { "name": "department", "description": "Which department the document belongs to", "output_type": "string", }, { "name": "Document Date", "description": "The date the document states about itself", "output_type": "date", }, ]: client.tags.upsert(tag_data=tag) job_id = str(uuid.uuid4()) client.metadata.generate.generate_batch( data_connector_name="raw-vectors", tag_names=["document_type", "department", "Document Date"], job_id=job_id, ) while True: progress = client.task_status.get_status(job_id=job_id) if progress.status in ("completed", "failed", "aborted"): break print(f" Classification {progress.percent_complete:.0f}%...") time.sleep(10) print(f"✓ Classification {progress.status}") ``` ## Step 3. Curate the Serving Slice A slice is a use case. Scope the serving collection to what retrieval should actually see, and exclude anything carrying a `Data Quality Status` flag: documents tagged `redundant` by the platform's duplicate detection, `expired` by a freshness rule, or otherwise flagged as unfit. See [Prepare an AI-Ready Dataset](/cookbooks/data-quality) for the full readiness flow. ```python theme={"dark"} serving_slice = client.data_slice.create( data_connector_name="raw-vectors", dataslice_name="rag-serving-set", description="Curated set for the RAG serving collection.", condition={ "condition": "AND", "children": [ {"tag": {"name": "document_type", "operator": "in", "values": ["contract", "policy", "report", "manual"]}}, {"tag": {"name": "Data Quality Status", "operator": "not_exists"}}, ], }, ) print(f"✓ Serving slice: {serving_slice.dataslice_id}") ``` ## Step 4. Export to the Serving Collection ```python theme={"dark"} client.data_slice.export_vdb( target_data_connector_name="serving-vectors", ori_data_connector_name="raw-vectors", dataslice_id=serving_slice.dataslice_id, export_level="chunk", ) print("✓ Serving collection populated") ``` Point your RAG pipeline at `ai_ready_documents`. Every chunk carries its tags, so retrieval can filter by `document_type`, `department`, or any tag you add. See [Scope Chatbot Answers with Metadata](/cookbooks/metadata-filtered-rag) for the retrieval side. ## Step 5. Maintain It Over Time As the raw collection grows, keep the serving collection current: classify new content on a nightly cadence, then re-export the slice. ```python theme={"dark"} client.workflows.upsert( workflow={ "name": "Nightly maintain: rag serving set", "description": "Classify new content and refresh the serving collection every night", "cadence": "0 0 * * *", "stages": [ {"jobs": [{"endpoint": "/classify_bulk", "endpoint_request_body": {"data_connector_name": "raw-vectors"}}]}, {"jobs": [{"endpoint": "/dataslice/export/vdb", "endpoint_request_body": { "target_data_connector_name": "serving-vectors", "ori_data_connector_name": "raw-vectors", "dataslice_id": serving_slice.dataslice_id, "export_level": "chunk"}}]}, ], }, ) print("✓ Nightly maintenance scheduled") ``` Combine with [Keep Answers Current Over Time](/cookbooks/freshness-curation) so expired documents drop out of the serving slice automatically, and answers never degrade as content ages. ## Next Steps Use the exported tags as retrieval filters. Keep expired content out of the serving set. Enrich document libraries at the source. All sources, destinations, and file types in one place. # Mask Sensitive Text Before It Reaches an LLM Source: https://docs.deasylabs.com/cookbooks/redact-sensitive-text Use detected sensitive values to redact text client-side, without withholding the whole document Sometimes a document should not be withheld entirely, only the sensitive parts inside it. A support ticket might be fine for an agent to read except for the customer's credit card number. This cookbook takes the values Deasy Labs already detected and masks them out of the text your application sends to an LLM, using plain Python. ## The Flow ```mermaid theme={"dark"} flowchart LR RUN[Classification job
sensitivity strategy] --> VAL[Matched values
metadata.list] TXT[Document text
data_source.document_text] --> MASK[Custom Python
str.replace] VAL --> MASK MASK --> LLM[LLM prompt · agent context] ``` ## Step 1. Run Sensitivity Detection This is the same detection job used in [Protect Sensitive Data](/cookbooks/pii-detection). Pick the classifiers whose matched values you want to mask, not the rollup tags: rollup tags like `PII (Personally Identifiable Information)` resolve to `true`/`false` and carry no literal value to redact. ```python theme={"dark"} import time import uuid from deasy import DeasyClient client = DeasyClient( base_url="https://unstructured.your-company.com/rest/unstructured", username="your-username", password="your-password", ) CONNECTOR = "my-sharepoint" SENSITIVITY_TAGS = ["Social Security Number", "Credit Card", "Email Address", "Phone Number"] job_id = str(uuid.uuid4()) client.metadata.generate.generate_batch( data_connector_name=CONNECTOR, tag_names=SENSITIVITY_TAGS, job_id=job_id, ) while True: progress = client.task_status.get_status(job_id=job_id) if progress.status in ("completed", "failed", "aborted"): break time.sleep(10) ``` ## Step 2. Read Back the Matched Values When a sensitivity tag has no fixed set of `available_values`, the matched text itself becomes the tag's value. Collect those values per file. ```python theme={"dark"} results = client.metadata.list( data_connector_name=CONNECTOR, tag_names=SENSITIVITY_TAGS, ) matches_by_file = {} for file_name, tags in (results.metadata or {}).items(): matched = [] for tag_name in SENSITIVITY_TAGS: tag = tags.get(tag_name) if tag and tag.file_level and tag.file_level.values: matched.extend(str(value) for value in tag.file_level.values) if matched: matches_by_file[file_name] = matched ``` ## Step 3. Mask the Matched Values This part is plain Python, nothing Deasy-specific. Replace the longest values first, so a short match inside a longer one (an area code inside a full phone number, for instance) does not leave a partial value behind. ```python theme={"dark"} def redact(text, matched_values, placeholder="[REDACTED]"): for value in sorted(set(matched_values), key=len, reverse=True): text = text.replace(value, placeholder) return text ``` ## Step 4. Mask Before You Prompt Pull the document's own text through the SDK rather than re-reading it from the source yourself. `document_text` returns each file's content keyed by node, since a file is chunked into nodes for retrieval. ```python theme={"dark"} text_response = client.data_source.document_text( data_connector_name=CONNECTOR, file_names=[file_name], ) safe_nodes = { node_id: redact(text, matches_by_file.get(file_name, [])) for node_id, text in text_response.file_to_nodes_to_text[file_name].items() } response = your_llm_client.complete(prompt="\n".join(safe_nodes.values())) ``` ## A Worked Example Say the source file is a support ticket short enough to be a single node. This is the text `document_text` returns for it, before any masking: ```text theme={"dark"} Hi, I'm locked out of my account. My SSN on file is 123-45-6789 and my card is 4111 1111 1111 1111. You can reach me at jane.doe@example.com or 415-555-0192 if you need to verify my identity. ``` After Step 1 and Step 2, `matches_by_file["ticket-4821.txt"]` holds the literal values the classification job found in that file: ```python theme={"dark"} ["123-45-6789", "4111 1111 1111 1111", "jane.doe@example.com", "415-555-0192"] ``` Running `redact(text, matches_by_file["ticket-4821.txt"])` from Step 3 on that node's text produces: ```text theme={"dark"} Hi, I'm locked out of my account. My SSN on file is [REDACTED] and my card is [REDACTED]. You can reach me at [REDACTED] or [REDACTED] if you need to verify my identity. ``` Only the four matched values changed. Everything else in the ticket, the parts an agent needs to actually help the customer, is untouched. If the classification job did not catch a value (a typo'd SSN, a phone number in an unexpected format), it will not be in the list and will not be masked. ## How to Use This * **For inline masking, not exclusion.** Use this when a document is mostly fine to share and only specific values need to be hidden. To keep a whole file out of AI systems, gate a slice instead, as in [Protect Sensitive Data](/cookbooks/pii-detection). * **Pick literal-value tags.** Only classifiers without a fixed `available_values` list return the matched text as their value. Rollup tags (`PII`, `PCI`, `PHI`) are booleans and have nothing to replace. * **Re-run after re-classification.** Matched values reflect the last classification job. If the source document changes, re-run detection before trusting the redaction. ## Next Steps Gate an entire slice on sensitivity tags instead of masking inline. Add custom identifiers to the sensitivity catalog before you redact them. How tag values and evidence are stored per file. Combine masking with the quality dimensions of the AI-ready gate. # Organize a SharePoint Library Source: https://docs.deasylabs.com/cookbooks/sharepoint-to-sharepoint Enrich a SharePoint library at the source with AI-extracted metadata columns and keep it maintained over time Enrich at the source. This cookbook connects a SharePoint document library, tags its documents with AI-extracted metadata, and writes the results back to the same library as native SharePoint columns. Users keep working where they always have, but now they can filter, sort, and search by contract type, dates, and value. A scheduled workflow keeps the columns current as documents change. ## What You'll Build ```mermaid theme={"dark"} flowchart LR SP[SharePoint Library
documents] --> DL[Deasy Labs
ingest + classify] DL -->|column_store export| SP2[Same SharePoint Library
+ metadata columns] CRON[Nightly workflow] -.->|maintain| DL ``` ## Prerequisites * A SharePoint site with a documents library and an Azure app registration (client ID, client secret, tenant ID) * Python 3.9+ ```bash theme={"dark"} pip install deasy_sdk-*.whl ``` ## Step 1. Connect the SharePoint Source ```python theme={"dark"} from deasy import DeasyClient client = DeasyClient( base_url="https://unstructured.your-company.com/rest/unstructured", username="your-username", password="your-password", ) source = client.data_source.create( connector_name="legal-library", connector_body={ "type": "SharepointDataSourceManager", "name": "legal-library", "client_id": "YOUR_CLIENT_ID", "client_secret": "YOUR_CLIENT_SECRET", "tenant_id": "YOUR_TENANT_ID", "sharepoint_site_name": "LegalDocuments", }, ) print(f"✓ Connected source: {source.profile_id}") ``` ## Step 2. Ingest the Library Ingestion runs as a background job and automatically captures source metadata (`Last Modified`, `File Type`, `Created By`, `Folder Structure`) on every document. ```python theme={"dark"} import time import uuid ingest_job = str(uuid.uuid4()) client.data_source.ingest( data_connector_name="legal-library", job_id=ingest_job, ) while client.task_status.get_status(job_id=ingest_job).status == "in_progress": time.sleep(10) print("✓ Ingestion complete") ``` ## Step 3. Define the Columns You Want Each tag becomes a SharePoint column. Classification extracts the values with evidence and confidence. ```python theme={"dark"} for tag in [ { "name": "contract_type", "description": "Type of contract (NDA, MSA, SLA, Employment, etc.)", "output_type": "string", "available_values": ["NDA", "MSA", "SLA", "Employment", "Other"], }, { "name": "effective_date", "description": "When the contract becomes effective", "output_type": "date", }, { "name": "expiration_date", "description": "When the contract expires or terminates", "output_type": "date", }, { "name": "total_value", "description": "Total monetary value of the contract if specified", "output_type": "number", }, ]: client.tags.upsert(tag_data=tag) classify_job = str(uuid.uuid4()) client.metadata.generate.generate_batch( data_connector_name="legal-library", tag_names=["contract_type", "effective_date", "expiration_date", "total_value"], job_id=classify_job, ) while True: progress = client.task_status.get_status(job_id=classify_job) if progress.status in ("completed", "failed", "aborted"): break print(f" Classification {progress.percent_complete:.0f}%...") time.sleep(10) print(f"✓ Classification {progress.status}") ``` ## Step 4. Write the Columns Back The destination points at the same site. `column_store` creates one SharePoint column per exported tag. ```python theme={"dark"} client.destination.create( connector_name="legal-library-columns", connector_body={ "type": "SharepointNodeDestinationManager", "name": "legal-library-columns", "client_id": "YOUR_CLIENT_ID", "client_secret": "YOUR_CLIENT_SECRET", "tenant_id": "YOUR_TENANT_ID", "sharepoint_site_name": "LegalDocuments", "documents_library_folder_name": "Contracts", }, ) export_result = client.destination.export( destination_name="legal-library-columns", data_connector_name="legal-library", export_level="file", export_metadata=True, metadata_format="column_store", export_tags=["contract_type", "effective_date", "expiration_date", "total_value"], ) print(f"✓ Exported {export_result.success} files ({export_result.failed} failed)") ``` ## What Users See in SharePoint | Document | Contract Type | Effective Date | Expiration Date | Total Value | | :------------------- | :------------ | :------------- | :-------------- | :---------- | | Acme-NDA-2024.pdf | NDA | 2024-01-15 | 2026-01-15 | - | | ServiceAgreement.pdf | MSA | 2024-03-01 | 2027-03-01 | \$150,000 | | Employment-JDoe.pdf | Employment | 2024-02-01 | - | \$95,000 | Users can filter and sort by any column, build views like "Expiring This Quarter", set alerts, and use SharePoint's native search with metadata facets. ## Step 5. Maintain It Over Time New and changed documents should get columns without anyone re-running the flow. Schedule ingest and classify on a nightly cadence, the same shape the app's predefined workflows use, then re-export on your own trigger or a second workflow stage. ```python theme={"dark"} client.workflows.upsert( workflow={ "name": "Nightly maintain: legal-library", "description": "Ingest new documents and classify them every night", "cadence": "0 0 * * *", "stages": [ {"jobs": [{"endpoint": "/ocr/ingest", "endpoint_request_body": {"data_connector_name": "legal-library"}}]}, {"jobs": [{"endpoint": "/classify_bulk", "endpoint_request_body": {"data_connector_name": "legal-library"}}]}, ], }, ) print("✓ Nightly maintenance scheduled") ``` ## Next Steps Make an existing vector index AI-ready. Gate what gets exported with quality signals. Detect PII before enriching shared libraries. Everything about scheduled maintenance. # Open Source Credits Source: https://docs.deasylabs.com/essentials/open-source-mentions Libraries and tools that power Deasy We're proud to build on these amazing open source projects ## psycopg2-binary **License:** GNU Lesser General Public License (LGPL)\ **Repository:** [https://github.com/psycopg/psycopg2](https://github.com/psycopg/psycopg2)\ **License URL:** [https://github.com/psycopg/psycopg2/blob/master/LICENSE](https://github.com/psycopg/psycopg2/blob/master/LICENSE) **Important LGPL Notice:** This software uses psycopg2-binary without modification. The LGPL requires that we inform users that this component is licensed under the LGPL, and that users can obtain the original source code. You have the following rights regarding this component: * **Source Code Access:** The complete source code for psycopg2-binary is available at: [https://github.com/psycopg/psycopg2](https://github.com/psycopg/psycopg2) * **Modification Rights:** You may modify and replace this library component * **Reverse Engineering:** You may reverse engineer this product to the extent necessary to debug modifications to the LGPL-licensed component * **Redistribution:** You may redistribute the LGPL component under LGPL terms The LGPL license permits you to use, modify, and replace this library. Our software is designed to work with the standard version of this library through dynamic linking. ## certifi **License:** Mozilla Public License 2.0 (MPL 2.0)\ **Repository:** [https://github.com/certifi/python-certifi](https://github.com/certifi/python-certifi)\ **License URL:** [https://mozilla.org/MPL/2.0/](https://mozilla.org/MPL/2.0/) This software uses certifi without modification. ## tqdm **License:** Mozilla Public License 2.0 (MPL 2.0) and MIT\ **Repository:** [https://github.com/tqdm/tqdm](https://github.com/tqdm/tqdm)\ **License URL:** [https://mozilla.org/MPL/2.0/](https://mozilla.org/MPL/2.0/) This software uses tqdm without modification. ## axe\_core\@4.10.2 **License:** Mozilla Public License 2.0 (MPL 2.0)\ **Repository:** [https://github.com/dequelabs/axe-core](https://github.com/dequelabs/axe-core)\ **License URL:** [https://mozilla.org/MPL/2.0/](https://mozilla.org/MPL/2.0/) This software uses axe-core without modification. ## caniuse\_lite\@1.0.30001692 **License:** Creative Commons Attribution 4.0 International (CC-BY-4.0)\ **Repository:** [https://github.com/browserslist/caniuse-lite](https://github.com/browserslist/caniuse-lite)\ **License URL:** [https://creativecommons.org/licenses/by/4.0/](https://creativecommons.org/licenses/by/4.0/) This software uses caniuse-lite without modification. Attribution is provided as required by the license. **Source Code Availability:** The source code for all Mozilla Public License 2.0 components is available at the repositories linked above. You have the right to obtain, modify, and redistribute the source code of these components under the terms of the MPL 2.0 license. ## License Compliance This product complies with all applicable open source license requirements: * **Attribution:** All required copyright notices and license texts are provided above * **Source Access:** Links to original source repositories are provided for all components * **User Rights:** For copyleft licenses (MPL 2.0, LGPL), users retain all rights granted by those licenses * **No Additional Restrictions:** Our license terms do not impose additional restrictions that conflict with the open source license requirements ## Contact If you have questions about these open source licenses or need additional information about compliance, please contact our legal team. # Collibra Source: https://docs.deasylabs.com/integrations/collibra Catalog Deasy document sets and tag definitions in the Collibra platform Deasy Labs integrates with the Collibra platform so unstructured data becomes part of your governed data landscape. Two asset types carry the integration: **Deasy Document Sets**, which catalog curated sets of documents defined by tag filters, and **Tag definitions**, which catalog the metadata vocabulary itself in the Deasy Glossary. ## Deasy Document Sets A document set is a group of documents defined by a tag-filter condition, cataloged as a `Deasy Document Set` asset in Collibra's Data Sources domain. The asset carries profiling attributes computed from the underlying files and a deep link back to Deasy. Deasy Document Set asset in Collibra showing file formats, document types, folders, file count, tag filters, and a View files in Deasy link | Attribute | What it holds | | :---------------------------------- | :------------------------------------------------------------ | | **Tag Filters** | The defining condition, for example `"Author" = "Alex Lyons"` | | **Document Types** | The extracted document types present in the set | | **File Formats** | File formats present in the set | | **Common Folders / File Locations** | Where the files live in the source system | | **Earliest Creation Date** | Oldest document in the set | | **Number of Files** | Set size | | **URL** | "View files in Deasy", a deep link to the live set | Document sets follow Collibra's asset lifecycle (for example `Candidate`), so governance teams review and approve them like any other asset. ## Tag Definitions in the Deasy Glossary Every tag becomes a `Tag definition` asset in the Deasy Glossary domain, so the vocabulary used to classify unstructured data is itself governed. The asset carries the tag's full extraction instruction, its constraints, and its group (`OOB` for the platform's built-in tags). Tag definition asset in Collibra showing the Author tag's description, max values, and group | Attribute | What it holds | | :-------------- | :-------------------------------------------------- | | **Description** | The full extraction instruction the classifier uses | | **Max Values** | How many values the tag may return | | **Group** | The tag's group, `OOB` for built-in platform tags | Tag definitions can also flow the other way: standards defined in Collibra import into Deasy as tags, carrying their Collibra identity (`collibra_asset_id`, `collibra_domain_id`) and governance metadata (record codes, retention periods, dispositions) with them. See [Taxonomies and Tags](/concepts/taxonomies-tags). ## How the Integration Works Tag definitions land in the Deasy Glossary domain, whether they originate in Deasy or import from Collibra standards. Run classification and build the tag-filtered sets your use cases need. Every extraction carries values, evidence, and confidence. Curated sets publish to Collibra as Deasy Document Set assets with their profiling attributes and a live link back to the files in Deasy. ```mermaid theme={"dark"} flowchart LR COL[Collibra Standards
tag definitions] -->|import| DL[Deasy Labs
classify + curate] DL -->|catalog| DS[Deasy Document Sets
Data Sources domain] DL -->|catalog| TD[Tag definitions
Deasy Glossary domain] ``` ## Why It Matters Unstructured documents are invisible to most governance programs. With document sets and tag definitions cataloged, governance teams see what document collections exist, how they are defined, and which vocabulary describes them, with the same lifecycle, responsibilities, and audit trail as any governed asset, and one click back to the live data in Deasy. ## Next Steps All sources, destinations, and file types in one place. Tag definitions, governance metadata, and strategies. # Databricks Source: https://docs.deasylabs.com/integrations/databricks Deliver curated slices to Unity Catalog Volumes with metadata alongside The Databricks integration is available today through early access ahead of general availability. Reach out to [the Deasy Labs team](https://www.deasylabs.com) to get set up. Curated slices delivered to Unity Catalog Volumes, with Deasy metadata written alongside so governed teams can discover and query the data in place. # Microsoft Purview Source: https://docs.deasylabs.com/integrations/microsoft-purview Surface Deasy metadata in the Purview data map The Microsoft Purview integration is available today through early access ahead of general availability. Reach out to [the Deasy Labs team](https://www.deasylabs.com) to get set up. Deasy metadata surfaced in the Microsoft Purview data map, so unstructured document sets and their tags become part of your governed catalog. # Integrations Overview Source: https://docs.deasylabs.com/integrations/overview Every system Deasy Labs connects to: sources, destinations, file types, and ecosystem integrations in one place Deasy Labs sits between your document repositories and the systems that consume curated data. This page is the complete integration matrix: where documents come from, where enriched data goes, and which file types the platform processes. ## Data Sources Sources are where documents live. Connect them with a [Data Connector](/concepts/data-connectors); ingestion automatically captures source metadata (`Last Modified`, `File Type`, `Created By`, `Folder Structure`) on every document.
Amazon S3
Amazon S3
Bucket name, Access Key, Secret Key
Azure Blob Storage
Azure Blob Storage
Account, Container, Credentials
Google Cloud Storage
Google Cloud Storage
Bucket, Service Account
SharePoint
SharePoint
Client ID/Secret, Tenant ID, Site Name
OneDrive
OneDrive
Client ID/Secret, Tenant ID, Site
PostgreSQL
PostgreSQL (pgvector)
Host URL, Database, Credentials, Port
Qdrant
Qdrant
URL, API Key, Collection
## Destinations Destinations receive enriched documents and metadata. Configure them in the app's Export Destinations tab or through the SDK's [destination resource](/concepts/destinations).
SharePoint
SharePoint
Metadata as native library columns, enrich at the source
OneDrive
OneDrive
Enriched documents and metadata in drive libraries
PostgreSQL
PostgreSQL
Metadata rows for analytics and hybrid search
Amazon S3
Amazon S3
Curated documents and metadata objects
Google Cloud Storage
Google Cloud Storage
Curated documents and metadata objects
Qdrant
Qdrant
Chunk-level records with tags as payload, via slice export
## Ecosystem Deasy document sets and tag definitions, cataloged for governance. Feed RAG pipelines with curated, metadata-rich chunks. ## Private Preview
Databricks
Databricks Private Preview
Curated slices delivered to Unity Catalog Volumes with metadata alongside
Microsoft Purview
Microsoft Purview Private Preview
Deasy metadata surfaced in the Purview data map
## Supported File Types | Category | Extensions | | :------------ | :-------------------------------------------------------- | | Documents | `.pdf`, `.docx`, `.rtf` | | Spreadsheets | `.xls`, `.xlsx`, `.xlsm`, `.xlsb`, `.odf`, `.ods`, `.odt` | | Presentations | `.ppt`, `.pptx` | | Email | `.msg` | | Web | `.html`, `.htm`, `.aspx`, `.xml` | | Data | `.json`, `.csv` | | Plain text | `.txt`, `.log` | ## Next Steps Create and manage source connections with the SDK. Export options, levels, and metadata formats. # Introduction Source: https://docs.deasylabs.com/introduction The right slice of your organization's knowledge, ready for AI in minutes **Deasy Labs** delivers the right slice of your organization's knowledge, ready for AI in minutes. It turns unstructured data, from a sprawling SharePoint to decades of PDFs, into the exact dataset your team is building against. Connect your storage, tag thousands of files per minute with AI-extracted metadata, slice your data any way you want, and deliver AI-ready datasets to vector databases, SharePoint, SQL warehouses, and the Collibra platform. ## How It Works
Data Sources
Amazon S3 SharePoint OneDrive PostgreSQL Qdrant
→
Connect & Ingest
OCR + source metadata
→
Tag
AI metadata extraction
→
Slice
curated datasets
→
Deliver
SharePoint Azure SQL Amazon S3 Qdrant
Maintained over time: [Workflows](/concepts/workflows) re-ingest, re-classify, and re-export on a schedule
Point a [Data Connector](/concepts/data-connectors) at your document store. No data migration needed. Define what you want to know about your documents with [Tags and Taxonomies](/concepts/taxonomies-tags), or let AI suggest them. Classification generates [Metadata](/concepts/metadata) with values, evidence, and confidence for every document. Build [Data Slices](/concepts/data-slices) that capture the exact subset each use case needs. Export slices to [Destinations](/concepts/destinations) and enrich documents at the source. Schedule [Workflows](/concepts/workflows) so datasets stay fresh as documents change. ## Start Building Go from zero to extracted metadata in under 5 minutes with the Python SDK. What teams build: RAG context, compliance, document management, governance. End-to-end recipes for RAG pipelines, PII detection, and data quality. Every endpoint, generated from the live OpenAPI specification. ## Why Teams Use It Curate before you compute. Retrieval works on high-quality, relevant knowledge instead of raw document dumps. Detect PII, PHI, and PCI automatically at scale and route sensitive content before it reaches downstream systems. Scheduled workflows keep datasets fresh, so answers stay grounded in current documents. # Quickstart Source: https://docs.deasylabs.com/quickstart Go from zero to extracting metadata in under 5 minutes Get started with Deasy Labs by running this complete example. You'll connect to a data source, define what metadata to extract, and see results in minutes. ## Installation Download and install the Python SDK wheel file: Contact us to get the SDK wheel file ```bash theme={"dark"} pip install deasy_sdk-*.whl ``` ## Supported Filetypes Deasy Labs can process a wide variety of document formats: .pdf, .docx, .rtf .xls, .xlsx, .xlsm, .xlsb, .odf, .ods, .odt .ppt, .pptx .msg .html, .htm, .aspx, .xml .json, .csv .txt, .log ## Language Support | Capability | Supported Languages | | :---------------------- | :------------------------------------------- | | **Document Processing** | Multilingual (all UTF-8 supported languages) | | **LLM Classification** | Multilingual (dependent on model) | | **PII Detection** | English only | ## Complete Example Copy and run this script to extract metadata from your documents: ```python theme={"dark"} import time import uuid from deasy import DeasyClient # 1. Initialize the client (basic auth) client = DeasyClient( base_url="https://unstructured.your-company.com/rest/unstructured", username="your-username", password="your-password", ) # Alternatively, authenticate with an API token issued from the web UI: # client = DeasyClient( # base_url="https://unstructured.your-company.com/rest/unstructured", # api_token="your-api-token", # user_id="your-username", # sent as the X-User-ID header # ) # 2. Create a data connector (S3 example) connector = client.data_source.create( connector_name="my-s3-bucket", connector_body={ "type": "S3DataSourceManager", "name": "my-s3-bucket", "bucket_name": "my-documents", "aws_access_key_id": "YOUR_ACCESS_KEY", "aws_secret_access_key": "YOUR_SECRET_KEY", "region": "us-east-1", }, ) print(f"✓ Created connector: {connector.profile_id}") # 3. Define the tags you want to extract for tag in [ { "name": "document_type", "description": "Type of document (invoice, contract, report, etc.)", "output_type": "string", "available_values": ["invoice", "contract", "report", "other"], }, { "name": "summary", "description": "A brief 2-3 sentence summary of the document", "output_type": "string", }, { "name": "key_date", "description": "The most important date mentioned in the document", "output_type": "date", }, ]: client.tags.upsert(tag_data=tag) print("✓ Created tags: document_type, summary, key_date") # 4. Extract metadata from all documents (runs as a background job) job_id = str(uuid.uuid4()) client.metadata.generate.generate_batch( data_connector_name="my-s3-bucket", tag_names=["document_type", "summary", "key_date"], job_id=job_id, ) while True: progress = client.task_status.get_status(job_id=job_id) print(f" Classification {progress.percent_complete:.0f}% complete") if progress.status in ("completed", "failed", "aborted"): break time.sleep(10) # 5. View the results results = client.metadata.list(data_connector_name="my-s3-bucket") for file_name, tags in results.metadata.items(): print(f"\nFile: {file_name}") for tag_name, data in tags.items(): if data.file_level: print(f" {tag_name}: {data.file_level.values}") ``` **About client configuration** * **`base_url`** is required and points to your Deasy deployment (e.g. `https://unstructured.your-company.com/rest/unstructured`). There is no default. * Any constructor argument can be set via an environment variable instead: `UNSTRUCTURED_CLIENT_BASE_URL`, `UNSTRUCTURED_USERNAME`, `UNSTRUCTURED_PASSWORD`, `UNSTRUCTURED_API_TOKEN`, `UNSTRUCTURED_USER_ID`. * **API tokens** are long-lived and issued from your Deasy deployment's web UI. The SDK does not create, refresh, or revoke them. ## What Just Happened? The Data Connector established a secure connection to your S3 bucket, allowing the platform to read your documents. The Tags told the platform what information to look for: document type, summary, and key date. Tags can be grouped into hierarchical Taxonomies for conditional extraction. The platform's AI analyzed each document in a background job (tracked via `task_status`) and extracted the structured metadata you defined. ## Next Steps Learn how Data Connectors, Taxonomies, and Metadata work together. Export enriched metadata to SharePoint. Explore all available endpoints and SDK methods. Set up sensitive data detection for compliance. # Use Cases Source: https://docs.deasylabs.com/use-cases What teams build with Deasy Labs, and the cookbooks that get you there A slice is a use case: every application gets exactly the documents it should see, gated by quality, freshness, and sensitivity. These are the four patterns teams build, each with runnable cookbooks. ## RAG and Agent Context An agent is only as accurate as the context it can reach, and raw document dumps fail it in predictable ways: duplicate copies, stale versions, lookalike chunks from the wrong documents, no access boundaries. These are metadata problems, not model problems. Deasy Labs gives you two layers of control over what the agent accesses: * **Curate once.** A [Data Slice](/concepts/data-slices) is the agent's vetted context: deduplicated, current, relevance-filtered, sensitivity-gated. In internal benchmarks, indexing only the latest version of each document improved retrieval accuracy by 46%. * **Route at runtime.** Within the slice, every chunk carries its tags. The agent routes by metadata, filtering to the right documents before similarity runs, and you can enforce metadata rules so anything tagged sensitive is never touched. In practice: for "what are the termination terms in our supplier agreements", the slice already excludes drafts, duplicates, and PII; `Contract Type = Supplier Agreement` narrows tens of thousands of chunks to a few hundred; similarity ranks within them; and the answer cites values that carry evidence and confidence. [Workflows](/concepts/workflows) keep the slice current as documents change. The curation flow that produces the agent's slice. Serve a curated collection from an existing one, maintained nightly. Tags as retrieval filters, end to end. Freshness rules keep the slice current. ## Document Management Not every consumer of metadata is an AI system. Enrich libraries at the source: extracted metadata lands as native SharePoint columns, so people filter, sort, and search where they already work, and a nightly workflow keeps the columns current. Connect, classify, write columns back, schedule maintenance. Let AI design the columns from your actual documents. ## Governance for Unstructured Data Document collections become visible to governance: curated sets are cataloged in Collibra as Deasy Document Set assets with profiling attributes, lifecycle status, and a deep link back to the live files, while the tag vocabulary itself is governed in the Deasy Glossary. Document sets and tag definitions, cataloged with relations. Governance metadata on tags: record codes, retention, dispositions. ## Compliance and Sensitive Data The platform's built-in classifiers detect PII, PHI, and PCI at scale, write the findings into rule-based tags, and route flagged documents to a restricted review path with an auditable metadata trail. Customer-facing AI systems retrieve only from the clean slice; sensitivity is an opt-in gate chosen per use case. Built-in classifiers, rule-based tags, and the gated slice. Enable sensitivity detection per workspace.