Skip to content

CX2 and NDEx Publishing

This document describes the current dismech workflow for exporting disorder pathographs to CX2 and publishing them to NDEx.

Overview

The CX2 export is built from the same pathograph graph used in the HTML disorder pages. It preserves all pathograph node types, including:

  • pathophysiology and other event nodes
  • phenotypes
  • treatments
  • experimental models
  • environmental and biochemical nodes
  • genetic and variant nodes

The exporter also adds NDEx-oriented metadata:

  • disease-level MONDO label on the network
  • NDEx-style top-level disease, reference, and tissue network attributes
  • NDEx/iQuery-friendly typing for gene-referencing nodes
  • ontology links on event nodes for CL, GO, UBERON, and related terms when present in the source YAML

Current Commands

Export one disorder to CX2 JSON:

just export-cx2 kb/disorders/Stargardt_Disease.yaml -o /tmp/Stargardt_Disease.cx2.json

Export all disorders to a directory:

just export-cx2-all -o /tmp/cx2

Upload one disorder directly to the NDEx test server:

just upload-cx2-test kb/disorders/Stargardt_Disease.yaml

Upload all disorders directly to the NDEx test server:

just upload-cx2-test-all

Important Behavior

just upload-cx2-test and just upload-cx2-test-all do not read previously exported JSON files. They re-run the exporter from the disorder YAML files and upload directly from that fresh CX2 output.

The test-server upload targets also pass --ndex-replace-existing by default. That means they look for existing networks on the same NDEx account with the same network name, update the newest match in place, and delete older same-name duplicates.

This means:

  • run just export-cx2-all -o /tmp/cx2 only when you want local JSON files for spot-checking
  • run just upload-cx2-test-all when you want to publish the current exporter output
  • you do not need to export first before uploading

Disorders with no pathograph edges are skipped rather than treated as errors.

NDEx Visibility

The current default upload visibility is PRIVATE. Publication should be a separate, explicit operation after the uploaded network has passed NDEx's server-side validation.

That default applies to:

  • uv run dismech-cx2 ... --ndex-upload
  • just upload-cx2-test
  • just upload-cx2-test-all

To override for a one-off upload:

just upload-cx2-test kb/disorders/Stargardt_Disease.yaml --visibility PUBLIC

Test Server Host

The just upload targets default to:

https://test.ndexbio.org

You can override that host with:

export NDEX_TEST_HOST=https://test.ndexbio.org

The uploader prints a viewer URL on the same host it uploaded to.

Credentials and Authentication

The current uploader uses NDEx username/password authentication through the ndex2 Python client. It does not use Google OAuth.

Set credentials in the shell before upload:

export NDEX_USERNAME=...
export NDEX_PASSWORD=...

Important operational notes:

  • A browser login via Google does not automatically make the CLI uploader work.
  • The NDEx API path used here expects username/password credentials.
  • On NDEx, the API username may be the NDEx account name rather than your email address.
  • The public NDEx server account and the test.ndexbio.org account may be different.

If test.ndexbio.org says your user does not exist, that usually means either:

  • you used an email address instead of the actual NDEx account name
  • or you do not have an account on the test server yet

What the CX2 Export Adds for NDEx

Network header attributes

The exporter now populates several NDEx-indexed network attributes that show up prominently in the NDEx info panel:

  • disease: built from the disorder-level disease_term, using the MONDO label as a link target
  • reference: built from top-level references when present, with fallback to evidence-level reference plus reference_title
  • tissue: built from ontology-backed locations and cell_types found across the disorder content

These are ordinary CX2 networkAttributes, not special NDEx-only aspects, but NDEx indexes and surfaces them by name.

Gene-aware iQuery typing

Nodes that reference human gene symbols get extra NDEx-friendly attributes so they can participate in iQuery/gene-centric search:

  • direct gene nodes may export as type=gene
  • descriptive event nodes that reference genes may export as type=proteinfamily plus member=[...]
  • the original DisMech semantic category is preserved separately as dismech_type

This keeps all pathograph nodes while still exposing the small gene-relevant subset in the form NDEx expects.

Event/pathophysiology nodes now expose linked ontology context when present in the YAML:

  • cell_type_ids, cell_type_urls, cell_type_links
  • biological_process_ids, biological_process_urls, biological_process_links
  • molecular_function_ids, molecular_function_urls, molecular_function_links
  • location_ids, location_urls, location_links

This is intended to make NDEx tables and property panels more useful without changing the visible node labels.

For a quick JSON check:

just export-cx2 kb/disorders/Stargardt_Disease.yaml -o /tmp/Stargardt_Disease.cx2.json

For a full publication pass to the NDEx test server:

export NDEX_USERNAME=...
export NDEX_PASSWORD=...
just upload-cx2-test-all

For debugging a failed upload, first try a single disorder:

just upload-cx2-test kb/disorders/Stargardt_Disease.yaml

Notes on Re-runs

Uploads create new networks. Re-running the same bulk upload will create duplicates unless you manually delete or replace the previous networks on NDEx.

The just upload-cx2-test and just upload-cx2-test-all wrappers now default to replacement mode, so this duplicate-creation behavior mainly applies when you invoke dismech-cx2 --ndex-upload directly without --ndex-replace-existing.

The uploader makes a best-effort attempt to set NDEx index_level=META after upload. If that post-upload call fails on the test server, the network upload itself is still treated as successful.

Production releases

Production publication uses the manually triggered Publish NDEx release GitHub Actions workflow. It is intentionally separate from pull-request and main-branch builds: merging a curation change must not mutate the public NDEx account.

The workflow has two jobs:

  1. export checks out the requested ref, generates a versioned CX2 corpus, and records hashes, counts, skipped disorders, export defects, and any UUIDs carried forward from a previous manifest.
  2. publish, when explicitly enabled, runs behind the protected ndex-production GitHub environment. It uploads to https://www.ndexbio.org, verifies the NDEx network summaries, and changes newly created or previously private networks to public only after the complete staged upload has succeeded. For a PUBLIC release, existing public networks remain public during a successful update; if one fails verification, that network is made private before the workflow stops. A PRIVATE release intentionally makes every network it touches private.

Configure these repository variables before running the workflow:

  • NDEX_AUTHOR
  • NDEX_RIGHTS
  • NDEX_RIGHTS_HOLDER

Configure NDEX_USERNAME and NDEX_PASSWORD as secrets on the protected ndex-production environment, and configure that environment with required reviewers. Environment-scoped secrets alone do not create an approval gate. The workflow never passes the password on a command line.

Production updates use the ndex_uuid values in a previous UUID registry. They do not search for or delete networks by name. After the first successful release, review uuid-registry.json from the verified artifact and commit it as conf/ndex-production-manifest.json before using update mode in a later release. This compact registry contains only slugs, UUIDs, and active or retired status; the full per-run manifest remains an Actions artifact.

When a slug in the previous registry is absent from the current knowledge base, the manifest reports it under retired_networks. Retirement is never automatic: an operator must decide whether to retain, privatize, or delete that NDEx network.

For the first production release, select the workflow's first_release input. That is the only workflow path that intentionally runs without a previous UUID registry, and it refuses to run if the configured registry already exists. Later releases require the configured registry path to exist; a missing or misspelled path fails rather than minting duplicate networks.

The workflow refuses to publish any new or changed orphan/unknown-node or missing-disease-metadata defect. Networks matching the exact reviewed backlog in conf/ndex-production-defect-allowlist.txt are quarantined: they remain in the manifest as SKIPPED_EXPORT_DEFECT, but are not uploaded. Fixing a quarantined network automatically makes it publishable; changing or introducing a defect fails the release audit until the finding is reviewed explicitly. The manifest therefore exposes both the release exclusions and the remaining remediation work without placing broken networks in production.

The manifest reports allowlist lines that no longer match a defect as unmatched_allowed_export_defects; remove those stale lines as the backlog is repaired. BLOCKED_EXPORT_DEFECT identifies a network with a new or changed finding. The manual --allow-export-defects CLI escape hatch suppresses that audit failure but still does not upload the defective network; if it already has an NDEx UUID, the registry preserves the mapping to the last verified network.

META indexing covers network attributes such as disease and tissue. Select ALL only when node/gene search is intended and its resource cost has been agreed with NDEx operators.

Recovering an interrupted release

The verified-artifact step runs even when publication fails, so an interrupted run retains manifest.json with every UUID minted before the failure. Download that artifact and derive a temporary recovery registry:

jq '{
  schema_version: "1.0",
  networks: ((
    [.networks[] | select(.ndex_uuid != null) |
      {slug, ndex_uuid, status: "ACTIVE"}] +
    [.retired_networks[] | select(.ndex_uuid != null) |
      {slug, ndex_uuid, status: "RETIRED"}]
  ) | sort_by(.slug))
}' manifest.json > conf/ndex-production-manifest.json

Review the recovered slug-to-UUID mapping, commit it on the ref that will be released, and rerun with first_release disabled. The next export carries those UUIDs forward and updates rather than duplicates the networks. After the run succeeds, replace the recovery registry with the verified uuid-registry.json artifact. A resumed public promotion skips networks already recorded as VERIFIED_PUBLIC and continues with the remaining private networks.

NDEx warnings remain fatal for production releases. This is deliberate: an operator should inspect and explicitly resolve a server warning rather than allowing the workflow to publish it silently.