Browse documentation
CROWDB / DOCUMENTATION

Iceberg tables

Supported catalog and FileIO behavior, Parquet scope, and the tested PyIceberg workflow.

DEVELOPMENT PREVIEW · OCTOBER 2, 2026

CROWDB implements an Iceberg REST catalog and native immutable table-file storage. A local PyIceberg workflow needs no separately deployed S3 object store.

The interface contract is independent of deployment. The example below uses the current single-node evaluation option; deployment and lifecycle details live in the deployment guides.

Single-node container

docker run -d --name crowdb-iceberg -p 127.0.0.1:9092:9092 crowdb/crowdb-iceberg:latest

Use Admin deployment for full volume, health, recovery, and limit details. Open the Console at http://127.0.0.1:9090/ with the 9090 mapping, then run:

docker exec crowdb-iceberg crowdb-monitor credentials show --format env

Set ICEBERG_URI and ICEBERG_TOKEN from the printed environment values.

Supported development surface

  • REST catalog configuration and namespace create/list/load/update/drop operations.
  • Table create/load/list, metadata updates, snapshots, and atomic commits with validation and retry/recovery authority.
  • Native immutable FileIO: upload, multipart upload, HEAD/range reads, and delegated scoped file credentials.
  • Parquet-backed selected-file validation. ORC selected-file validation remains a separate deferred feature.
  • PyIceberg create/append/scan with Arrow and pandas, including container service recovery and persisted-volume restart coverage.

The REST/official-SDK suites cover the declared protocol profile. They do not certify every operation of Spark, Flink, Trino, DuckDB, or every Iceberg format/version. Historical/batched/Polars and cross-engine container acceptance remain separate work.

Create, append, and scan

Set the printed ICEBERG_URI and ICEBERG_TOKEN in your client environment. The default URI is http://localhost:9092. The retained container client uses PyIceberg 0.11.1 with Arrow/pandas.

python -m pip install 'pyiceberg[pyarrow]==0.11.1' pandas
import os
import pandas as pd
import pyarrow as pa
from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "crowdb", type="rest",
    uri=os.environ["ICEBERG_URI"],
    token=os.environ["ICEBERG_TOKEN"],
)
catalog.create_namespace_if_not_exists("demo")
rows = pa.Table.from_pandas(
    pd.DataFrame({"order_id": [1, 2], "amount": [120, 80]}),
    preserve_index=False,
)
table = catalog.create_table("demo.orders", schema=rows.schema)
table.append(rows)
print(catalog.load_table("demo.orders").scan().to_pandas())

Use a new table name on repeated runs, or load an existing table before appending. The catalog response supplies native FileIO configuration and scoped credentials. Reload the table before a later write if its delegated grant has expired; the current profile uses 15-minute grants.

Client coverage

  • PyIceberg/Arrow/pandas: table creation, append, scans, and dataframe result checks in the container fixture. DuckDB 1.4.3 direct REST catalog reads are also verified.
  • TPC loader: TPC-H/TPC-DS Parquet import and row/query checks. See the loader guide; these are development checks, not TPC certification.
  • Spark, Flink, and Trino cross-engine acceptance are not claimed by this guide.

Limits and model boundaries

Iceberg owns tables, snapshots, commits, and file reachability independently of the general S3 namespace. The catalog and FileIO share port 9092; S3 port 9091 is not needed for this example. General S3 buckets cannot enumerate Iceberg table files.

Selected ORC validation, broad native FileIO prefix listing, Dataset access, and GPU delivery are outside this documented profile. The single-node image has no redundancy, no supported volume upgrade path, and disabled Iceberg physical GC. See deployment limits.