{"id":"dask","name":"dask","summary":"RAMより大きなPandas/NumPyワークフロー向けの分散コンピューティング。既存のpandaやNumPyコードをメモリやクラスター間でスケールさせる必要がある場合に使ってください。","body":"# Dask\n\n## Overview\n\nDask is a Python library for parallel and distributed computing that enables three critical capabilities:\n- **Larger-than-memory execution** on single machines for data exceeding available RAM\n- **Parallel processing** for improved computational speed across multiple cores\n- **Distributed computation** supporting terabyte-scale datasets across multiple machines\n\nDask scales from laptops (processing ~100 GiB) to clusters (processing ~100 TiB) while maintaining familiar Python APIs.\n\n**Current upstream:** dask **2026.3.0** (PyPI, March 2026). Docs: [docs.dask.org](https://docs.dask.org/en/stable/). Since **2025.1.0**, the expression-based DataFrame API with query planning is the only implementation — do not install `dask-expr` separately or set `dataframe.query-planning: False`.\n\n## Quick Start\n\n### Installation\n\n```bash\nuv pip install \"dask>=2025.1\"\n```\n\nFor a typical pandas/NumPy workflow with the distributed scheduler and dashboard:\n\n```bash\nuv pip install \"dask[complete]\"\n```\n\nRemote object storage (S3, GCS, Azure):\n\n```bash\nuv pip install s3fs    # s3:// paths\nuv pip install gcsfs   # gs:// paths\n```\n\nRequires **Python 3.10+** (3.9 support dropped in 2024.12). DataFrame I/O requires **PyArrow 16+** (as of dask 2026.1.2).\n\n## When to Use This Skill\n\nThis skill should be used when:\n- Process datasets that exceed available RAM\n- Scale pandas or NumPy operations to larger datasets\n- Parallelize computations for performance improvements\n- Process multiple files efficiently (CSVs, Parquet, JSON, text logs)\n- Build custom parallel workflows with task dependencies\n- Distribute workloads across multiple cores or machines\n\n## Core Capabilities\n\nDask provides five main components, each suited to different use cases:\n\n### 1. DataFrames - Parallel Pandas Operations\n\n**Purpose**: Scale pandas operations to larger datasets through parallel processing.\n\n**When to Use**:\n- Tabular data exceeds available RAM\n- Need to process multiple CSV/Parquet files together\n- Pandas operations are slow and need parallelization\n- Scaling from pandas prototype to production\n\n**Reference Documentation**: For comprehensive guidance on Dask DataFrames, refer to `references/dataframes.md` which includes:\n- Reading data (single files, multiple files, glob patterns)\n- Common operations (filtering, groupby, joins, aggregations)\n- Custom operations with `map_partitions`\n- Performance optimization tips\n- Common patterns (ETL, time series, multi-file processing)\n\n**Quick Example**:\n```python\nimport dask.dataframe as dd\n\n# Read multiple files as single DataFrame\nddf = dd.read_csv('data/2024-*.csv')\n\n# Operations are lazy until compute()\nfiltered = ddf[ddf['value'] > 100]\nresult = filtered.groupby('category').mean().compute()\n```\n\n**Key Points**:\n- Operations are lazy (build task graph) until `.compute()` called\n- Use `map_partitions` for efficient custom operations\n- Convert to DataFrame early when working with structured data from other sources\n\n### 2. Arrays - Parallel NumPy Operations\n\n**Purpose**: Extend NumPy capabilities to datasets larger than memory using blocked algorithms.\n\n**When to Use**:\n- Arrays exceed available RAM\n- NumPy operations need parallelization\n- Working with scientific datasets (HDF5, Zarr, NetCDF)\n- Need parallel linear algebra or array operations\n\n**Reference Documentation**: For comprehensive guidance on Dask Arrays, refer to `references/arrays.md` which includes:\n- Creating arrays (from NumPy, random, from disk)\n- Chunking strategies and optimization\n- Common operations (arithmetic, reductions, linear algebra)\n- Custom operations with `map_blocks`\n- Integration with HDF5, Zarr, and XArray\n\n**Quick Example**:\n```python\nimport dask.array as da\n\n# Create large array with chunks\nx = da.random.random((100000, 100000), chunks=(10000, 10000))\n\n# Operations are lazy\ny = x + 100\nz = y.mean(axis=0)\n\n# Compute result\nresult = z.compute()\n```\n\n**Key Points**:\n- Chunk size is critical (aim for ~100 MB per chunk)\n- Operations work on chunks in parallel\n- Rechunk data when needed for efficient operations\n- Use `map_blocks` for operations not available in Dask\n\n### 3. Bags - Parallel Processing of Unstructured Data\n\n**Purpose**: Process unstructured or semi-structured data (text, JSON, logs) with functional operations.\n\n**When to Use**:\n- Processing text files, logs, or JSON records\n- Data cleaning and ETL before structured analysis\n- Working with Python objects that don't fit array/dataframe formats\n- Need memory-efficient streaming processing\n\n**Reference Documentation**: For comprehensive guidance on Dask Bags, refer to `references/bags.md` which includes:\n- Reading text and JSON files\n- Functional operations (map, filter, fold, groupby)\n- Converting to DataFrames\n- Common patterns (log analysis, JSON processing, text processing)\n- Performance considerations\n\n**Quick Example**:\n```python\nimport dask.bag as db\nimport json\n\n# Read and parse JSON files\nbag = db.read_text('logs/*.json').map(json.loads)\n\n# Filter and transform\nvalid = bag.filter(lambda x: x['status'] == 'valid')\nprocessed = valid.map(lambda x: {'id': x['id'], 'value': x['value']})\n\n# Convert to DataFrame for analysis\nddf = processed.to_dataframe()\n```\n\n**Key Points**:\n- Use for initial data cleaning, then convert to DataFrame/Array\n- Use `foldby` instead of `groupby` for better performance\n- Operations are streaming and memory-efficient\n- Convert to structured formats (DataFrame) for complex operations\n\n### 4. Futures - Task-Based Parallelization\n\n**Purpose**: Build custom parallel workflows with fine-grained control over task execution and dependencies.\n\n**When to Use**:\n- Building dynamic, evolving workflows\n- Need immediate task execution (not lazy)\n- Computations depend on runtime conditions\n- Implementing custom parallel algorithms\n- Need stateful computations\n\n**Reference Documentation**: For comprehensive guidance on Dask Futures, refer to `references/futures.md` which includes:\n- Setting up distributed client\n- Submitting tasks and working with futures\n- Task dependencies and data movement\n- Advanced coordination (queues, locks, events, actors)\n- Common patterns (parameter sweeps, dynamic tasks, iterative algorithms)\n\n**Quick Example**:\n```python\nfrom dask.distributed import Client\n\nclient = Client()  # Create local cluster\n\n# Submit tasks (executes immediately)\ndef process(x):\n    return x ** 2\n\nfutures = client.map(process, range(100))\n\n# Gather results\nresults = client.gather(futures)\n\nclient.close()\n```\n\n**Key Points**:\n- Requires distributed client (even for single machine)\n- Tasks execute immediately when submitted\n- Pre-scatter large data to avoid repeated transfers\n- ~1ms overhead per task (not suitable for millions of tiny tasks)\n- Use actors for stateful workflows\n\n### 5. Schedulers - Execution Backends\n\n**Purpose**: Control how and where Dask tasks execute (threads, processes, distributed).\n\n**When to Choose Scheduler**:\n- **Threads** (default): NumPy/Pandas operations, GIL-releasing libraries, shared memory benefit\n- **Processes**: Pure Python code, text processing, GIL-bound operations\n- **Synchronous**: Debugging with pdb, profiling, understanding errors\n- **Distributed**: Need dashboard, multi-machine clusters, advanced features\n\n**Reference Documentation**: For comprehensive guidance on Dask Schedulers, refer to `references/schedulers.md` which includes:\n- Detailed scheduler descriptions and characteristics\n- Configuration methods (global, context manager, per-compute)\n- Performance considerations and overhead\n- Common patterns and troubleshooting\n- Thread configuration for optimal performance\n\n**Quick Example**:\n```python\nimport dask\nimport dask.dataframe as dd\n\n# Use threads for DataFrame (default, good for numeric)\nddf = dd.read_csv('data.csv')\nresult1 = ddf.mean().compute()  # Uses threads\n\n# Use processes for Python-heavy work\nimport dask.bag as db\nbag = db.read_text('logs/*.txt')\nresult2 = bag.map(python_function).compute(scheduler='processes')\n\n# Use synchronous for debugging\ndask.config.set(scheduler='synchronous')\nresult3 = problematic_computation.compute()  # Can use pdb\n\n# Use distributed for monitoring and scaling\nfrom dask.distributed import Client\nclient = Client()\nresult4 = computation.compute()  # Uses distributed with dashboard\n```\n\n**Key Points**:\n- Threads: Lowest overhead (~10 µs/task), best for numeric work\n- Processes: Avoids GIL (~10 ms/task), best for Python work\n- Distributed: Monitoring dashboard (~1 ms/task), scales to clusters\n- Can switch schedulers per computation or globally\n\n## Best Practices\n\nFor comprehensive performance optimization guidance, memory management strategies, and common pitfalls to avoid, refer to `references/best-practices.md`. Key principles include:\n\n### Start with Simpler Solutions\nBefore using Dask, explore:\n- Better algorithms\n- Efficient file formats (Parquet instead of CSV)\n- Compiled code (Numba, Cython)\n- Data sampling\n\n### Critical Performance Rules\n\n**1. Don't Load Data Locally Then Hand to Dask**\n```python\n# Wrong: Loads all data in memory first\nimport pandas as pd\ndf = pd.read_csv('large.csv')\nddf = dd.from_pandas(df, npartitions=10)\n\n# Correct: Let Dask handle loading\nimport dask.dataframe as dd\nddf = dd.read_csv('large.csv')\n```\n\n**2. Avoid Repeated compute() Calls**\n```python\n# Wrong: Each compute is separate\nfor item in items:\n    result = dask_computation(item).compute()\n\n# Correct: Single compute for all\ncomputations = [dask_computation(item) for item in items]\nresults = dask.compute(*computations)\n```\n\n**3. Don't Build Excessively Large Task Graphs**\n- Increase chunk sizes if millions of tasks\n- Use `map_partitions`/`map_blocks` to fuse operations\n- Check task graph size: `len(ddf.__dask_graph__())`\n\n**4. Choose Appropriate Chunk Sizes**\n- Target: ~100 MB per chunk (or 10 chunks per core in worker memory)\n- Too large: Memory overflow\n- Too small: Scheduling overhead\n\n**5. Use the Dashboard**\n```python\nfrom dask.distributed import Client\nclient = Client()\nprint(client.dashboard_link)  # Monitor performance, identify bottlenecks\n```\n\n## Common Workflow Patterns\n\n### ETL Pipeline\n```python\nimport dask.dataframe as dd\n\n# Extract: Read data\nddf = dd.read_csv('raw_data/*.csv')\n\n# Transform: Clean and process\nddf = ddf[ddf['status'] == 'valid']\nddf['amount'] = ddf['amount'].astype('float64')\nddf = ddf.dropna(subset=['important_col'])\n\n# Load: Aggregate and save\nsummary = ddf.groupby('category').agg({'amount': ['sum', 'mean']})\nsummary.to_parquet('output/summary.parquet')\n```\n\n### Unstructured to Structured Pipeline\n```python\nimport dask.bag as db\nimport json\n\n# Start with Bag for unstructured data\nbag = db.read_text('logs/*.json').map(json.loads)\nbag = bag.filter(lambda x: x['status'] == 'valid')\n\n# Convert to DataFrame for structured analysis\nddf = bag.to_dataframe()\nresult = ddf.groupby('category').mean().compute()\n```\n\n### Large-Scale Array Computation\n```python\nimport dask.array as da\n\n# Load or create large array\nx = da.from_zarr('large_dataset.zarr')\n\n# Process in chunks\nnormalized = (x - x.mean()) / x.std()\n\n# Save result (use mode= for overwrite; zarr_array_kwargs for compression)\nda.to_zarr(normalized, 'normalized.zarr', mode='w')\n```\n\n### Custom Parallel Workflow\n```python\nfrom dask.distributed import Client\n\nclient = Client()\n\n# Scatter large dataset once\ndata = client.scatter(large_dataset)\n\n# Process in parallel with dependencies\nfutures = []\nfor param in parameters:\n    future = client.submit(process, data, param)\n    futures.append(future)\n\n# Gather results\nresults = client.gather(futures)\n```\n\n## Selecting the Right Component\n\nUse this decision guide to choose the appropriate Dask component:\n\n**Data Type**:\n- Tabular data → **DataFrames**\n- Numeric arrays → **Arrays**\n- Text/JSON/logs → **Bags** (then convert to DataFrame)\n- Custom Python objects → **Bags** or **Futures**\n\n**Operation Type**:\n- Standard pandas operations → **DataFrames**\n- Standard NumPy operations → **Arrays**\n- Custom parallel tasks → **Futures**\n- Text processing/ETL → **Bags**\n\n**Control Level**:\n- High-level, automatic → **DataFrames/Arrays**\n- Low-level, manual → **Futures**\n\n**Workflow Type**:\n- Static computation graph → **DataFrames/Arrays/Bags**\n- Dynamic, evolving → **Futures**\n\n## Integration Considerations\n\n### File Formats\n- **Efficient**: Parquet, HDF5, Zarr (columnar, compressed, parallel-friendly)\n- **Compatible but slower**: CSV (use for initial ingestion only)\n- **For Arrays**: HDF5, Zarr, NetCDF\n\n### Conversion Between Collections\n```python\n# Bag → DataFrame\nddf = bag.to_dataframe()\n\n# DataFrame → Array (for numeric data)\narr = ddf.to_dask_array(lengths=True)\n\n# Array → DataFrame\nddf = dd.from_dask_array(arr, columns=['col1', 'col2'])\n```\n\n### With Other Libraries\n- **XArray**: Wraps Dask arrays with labeled dimensions (geospatial, imaging)\n- **Dask-ML**: Machine learning with scikit-learn compatible APIs\n- **Distributed**: Advanced cluster management and monitoring\n\n## Debugging and Development\n\n### Iterative Development Workflow\n\n1. **Test on small data with synchronous scheduler**:\n```python\ndask.config.set(scheduler='synchronous')\nresult = computation.compute()  # Can use pdb, easy debugging\n```\n\n2. **Validate with threads on sample**:\n```python\nsample = ddf.head(1000)  # Small sample\n# Test logic, then scale to full dataset\n```\n\n3. **Scale with distributed for monitoring**:\n```python\nfrom dask.distributed import Client\nclient = Client()\nprint(client.dashboard_link)  # Monitor performance\nresult = computation.compute()\n```\n\n### Common Issues\n\n**Memory Errors**:\n- Decrease chunk sizes\n- Use `persist()` strategically and delete when done\n- Check for memory leaks in custom functions\n\n**Slow Start**:\n- Task graph too large (increase chunk sizes)\n- Use `map_partitions` or `map_blocks` to reduce tasks\n\n**Poor Parallelization**:\n- Chunks too large (increase number of partitions)\n- Using threads with Python code (switch to processes)\n- Data dependencies preventing parallelism\n\n## Reference Files\n\nAll reference documentation files can be read as needed for detailed information:\n\n- `references/dataframes.md` - Complete Dask DataFrame guide\n- `references/arrays.md` - Complete Dask Array guide\n- `references/bags.md` - Complete Dask Bag guide\n- `references/futures.md` - Complete Dask Futures and distributed computing guide\n- `references/schedulers.md` - Complete scheduler selection and configuration guide\n- `references/best-practices.md` - Comprehensive performance optimization and troubleshooting\n\nLoad these files when users need detailed information about specific Dask components, operations, or patterns beyond the quick guidance provided here.","author":"@K-Dense-AI","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/K-Dense-AI/scientific-agent-skills/tree/main/skills/dask","license":"MIT","category":"document","lang":"en","tokens":3439,"stars":0,"calls30d":1,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"references/arrays.md","size":11642,"sha256":"e4979c41f3f1ad18baccb4da3f8f339020ba5180af01fbd45ddeaf6a71194c6e"},{"path":"references/bags.md","size":10869,"sha256":"5ff94d547b99baa80fef1f9b645aeeb73ee3efbecc6bc85a3d36a97d5d6702c9"},{"path":"references/best-practices.md","size":7317,"sha256":"8a7ed9cdc0b38a2754fae5af72c88fc15cdc8fe15afc517a5b2c50ede50680ae"},{"path":"references/dataframes.md","size":9098,"sha256":"d8b30342db6b8ef4e8a2625579d8987b3f7e4eba3037971472c689491b3aea77"},{"path":"references/futures.md","size":12051,"sha256":"6f4382ecb29466973c95b3ea7e58441566f0d566b41ef89ea89c6959e3a30e61"},{"path":"references/schedulers.md","size":11865,"sha256":"97d442a33a9bb97842c4e5c21157058de09aa4079c752332a55f92edd70ed745"}],"requires":{"mcp":[],"tools":["Read Write Edit Bash"]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":["docs.dask.org"]}}