Transit 2050

HostnamesWithCube.txt

@@ -0,0 +1 @@
+localhost

Local Machine Instructions.md

@@ -0,0 +1,97 @@
+# Intro
+
+This code has been updated to run on a laptop that has Cube Voyager 6.5.0 installed locally. This is primarily the removal and override of anything that appears to be MTC specific (e.g. paths to cube hostnames and a batch file used when calling runtpp).
+
+Questions can be directed to Andrew Rohne at RSG (andrew.rohne (at) rsginc.com). 9/20/2023.
+
+---
+Note: This is still being written! 
+---
+
+# Setting Up Cube
+
+The computer's path needs to include the path to Cube Voyager to use an executable there. Follow [these instructions](https://www.computerhope.com/issues/ch000549.htm) for your OS version, add C:\Program Files\Citilabs\CubeVoyager to the path.
+
+To test this, open Anaconda Prompt or Anaconda Powershell and type `runtpp`. There should be an error that says "Error in Arg 1: (null)".
+
+# Setting Up Network Wrangler
+
+This largely follows [the Network Wrangler Documentation](https://github.com/BayAreaMetro/modeling-website/wiki/Network-Building-with-NetworkWrangler#step-4-build-a-network-with-your-project) but is distilled down to something a little more concise.
+
+1. Download the [Anaconda environment](https://github.com/RSGInc/NetworkWrangler/blob/transit_2050/environment_nw.yml) to a location on your hard drive.
+2. Created the local Anaconda environment. Open Anaconda Prompt or Anaconda Powershell (either will work, hereinafter referred to as Anaconda Prompt) and type:
+    `cd path\you\downloaded\the\above\file\to`
+    `conda env create -f environment_nw.yml`
+3. Switched to it. Use the following command in the Anaconda prompt
+    `conda activate NetworkWrangler`
+4. Change directories to an appropriate location on your hard drive.
+5. Clone the network Wrangler Repo
+    `git clone https://github.com/RSGInc/NetworkWrangler.git`
+6. Go into that folder and checkout the transit_2050 branch
+    `git checkout transit_2050`
+7. Install NetworkWrangler. 
+    `cd NetworkWrangler`
+    `pip install -e .`
+8. Open python, import Wrangler, ensure no errors
+9. Get the base network - clone https://github.com/BayAreaMetro/TM1_2015_Base_Network to a local folder (I used C:\Models). This probably should not be done inside the network wrangler folder.
+10. Get the projects from Box
+    Not-RSG: From https://mtcdrive.box.com/s/unic7tf0sokleacg4fgu0dtu8ixkyfg6
+    RSG: I Copied to the files folder to Sharepoint as "04_Network_Work\MTC Data\NetworkProjects_20230719.zip
+    I unzipped to C:\Models\TM1_NetworkProjects
+11. Also download everything on Box since July 19
+    Not-RSG: use box, sort by date, select all necessary
+    RSG: Copied to files folder as "04_Network_Work\MTC Data\TM1_NetworkProjects-selected.zip"
+12. Set base network env var. Use these commands in an Anaconda prompt:
+    `conda env config vars set "TM1_2015_Base_Network=C:\Models\TM1_2015_Base_Network"`
+    `conda env config vars set "TM1_NetworkProjects=C:\Models\TM1_NetworkProjects"`
+    `conda env config vars set "CUBE_HOST_FILE=C:\Models\NetworkWrangler\HostnamesWithCube.txt"`
+    `conda activate NetworkWrangler`
+
+    Note: you can test these using `conda env config vars list`
+    Note2: Don't forget to reset if you start editing project cards straight from Box.
+13. If you wish to use the 'reportDiff' function, you will need to install and activate ArcGIS Pro on your machine. The NetworkWrangler environment already has `arcpy` environment installed with it.
+# Running Network Wrangler for the Base Year
+1. Be in the correct Anaconda Environment (if not already)
+    `conda activate NetworkWrangler`
+2. Build the test network
+    `python .\build_network_mtc.py Test .\net_spec_test.py`\
+   (If 'reportDiff' is desired, add `--create_project_diffs` to the end of the command above.)
+
+# Test Project Coding 
+
+The basic version is to add the cards to the TM1_NetworkProjects folder, and then copy the net_spec_test.py script to something a little more descriptive (for the BART San Jose example, I used `net_spec_MAJ_BRT030001_BART_to_SanJose.py`). The script needs to have a project code, scenario (for the purposes of this project, I use `build`)
+
+Run with `python build_network_mtc.py build net_spec_MAJ_BRT030001_BART_to_SanJose.py`
+
+# Adding a Project Card for PPA
+
+Use [build_network_mtc_add_project.py](https://github.com/BayAreaMetro/NetworkWrangler/blob/master/scripts/build_network_mtc_add_project.py) for PPA since only a small number of projects are added (mostly just 1). The base network should be the latest 2050 network in [Sample of L drive Projects Folder](https://mtcdrive.box.com/s/vbpsrs7tpvj1qfxmasink52wls8pexrj), one for each future. The script will automatically use `XXX_17` (latest). In [build_network_mtc_add_project.py](https://github.com/BayAreaMetro/NetworkWrangler/blob/master/scripts/build_network_mtc_add_project.py), point `PPA_DIR` to [Sample of L drive Projects Folder](https://mtcdrive.box.com/s/vbpsrs7tpvj1qfxmasink52wls8pexrj) and `NODE_NAMES` to `TM1_2015_Base_Network\Node Description.xls`.
+
+Required arguments to use the script:
+
+1. Future scenario: one of these three - `CleanAndGreen`, `RisingTides`, and `BackToTheFuture`. `CleanAndGreen` is taken as an example to demonstrate.
+2. Input network: `..\Sample of L drive Projects Folder\2050_TM151_PPA_CG_17\INPUT`.
+3. Project to be added: one of the projects in `TM1_NetworkProjects`. Take `MAJ_BRT030001_BART_to_SanJose` as an example.
+4. Nature of project: `--hwy`, `--trn`, or both.
+5. Output: create a separate folder - `..\MTC_Outputs\MAJ_BRT030001_BART_to_SanJose_CG`.
+6. `project_short_id`: in this case it's `MAJ_BRT030001_BART_to_SanJose_CG`.
+
+Full example: `python build_network_mtc_add_project.py --trn --input_network "..\Box\Performance and Equity\Project Performance\Sample of L drive Projects Folder\2050_TM151_PPA_CG_17\INPUT" --output_network "..\MTC_Outputs\MAJ_BRT030001_BART_to_SanJose_CG" --create_project_diffs CleanAndGreen MAJ_BRT030001_BART_to_SanJose_CG MAJ_BRT030001_BART_to_SanJose`
+
+
+## Determine Pre-requisite Projects
+
+# Notes and Errors
+
+## Pre-requisite Projects
+
+If you see this: `WranglerLogger: DEBUG    !!!WARNING!!! Some PRE-REQUISITES were not found or ordered correctly.  Continue anyway? (y/n)`, that means you're missing one or more projects and these projects are required for the project you're using to work. Listed above this, you'll see the pre-requisite projects:
+
+```
+WranglerLogger: INFO     Requirement verification - Pre-requisite
+WranglerLogger: INFO         Year    Project                                                Pre-requisite Project                              Year
+WranglerLogger: INFO     trn 2020.trn.01 MAJ_BRT030001_BART_to_SanJose                      trn ALA050015_BART_to_WarmSprings                      -1
+WranglerLogger: INFO     trn 2020.trn.01 MAJ_BRT030001_BART_to_SanJose                      trn SCL110005_BART_to_Berryessa                        -1
+```
+
+In this case, it can be fixed by including the prerequisite projects in the appropriate network.

Wrangler/HighwayNetwork.py

@@ -1,4 +1,5 @@
 import collections, csv, os, re, shutil, subprocess, time
+import copy
 from socket         import gethostname, getfqdn
 
 from .HwySpecsRTP import HwySpecsRTP
@@ -31,7 +32,8 @@ def getCubeHostnames():
 
         # read them
         HighwayNetwork.cube_hostnames = []
-        f = open(r"Y:\COMMPATH\HostnamesWithCube.txt")
+        env = copy.copy(os.environ)
+        f = open(env['CUBE_HOST_FILE'])
         for line in f:
             if line[0] == "#": continue
             HighwayNetwork.cube_hostnames.append(line.split()[0])  # use the first token of non-comment lines
@@ -146,7 +148,7 @@ def applyProject(self, parentdir, networkdir, gitdir, projectsubdir=None, **kwar
 
             # dispatch it, cube license
             hostname = gethostname().lower()
-            if hostname not in HighwayNetwork.getCubeHostnames():
+            if False: #hostname not in HighwayNetwork.getCubeHostnames():
                 print("Dispatching cube script to taraval from %s".format(hostname))
                 f = open(os.path.join(applyDir,'runtpp_dispatch.tmp'), 'w')
                 f.write("runtpp " + applyScript + "\n")

Xdrive Script (1).txt

@@ -0,0 +1,14 @@
+Please copy the entire script below and paste into PowerShell.
+Do not run Powershell as Administrator.
+The Zdrive should persist with restarts of the computer.
+
+
+$connectTestResult = Test-NetConnection -ComputerName mtcproject.file.core.windows.net -Port 445
+if ($connectTestResult.TcpTestSucceeded) {
+    # Save the password so the drive will persist on reboot
+    cmd.exe /C "cmdkey /add:`"mtcproject.file.core.windows.net`" /user:`"localhost\mtcproject`" /pass:`"uaBceR+X4ssB5Uu3z25AzKCWvIWBn69JZ92OjMm7MWSVc2hG0MaXkKhYO0kdApUuxz/BBPcxNWR3+AStaJEEGg==`""
+    # Mount the drive
+    New-PSDrive -Name X -PSProvider FileSystem -Root "\\mtcproject.file.core.windows.net\zdrive" -Persist
+} else {
+    Write-Error -Message "Unable to reach the Azure storage account via port 445. Check to make sure your organization or ISP is not blocking port 445, or use Azure P2S VPN, Azure S2S VPN, or Express Route to tunnel SMB traffic over a different port."
+}

_static/Cube/CubeNet.py

@@ -22,8 +22,8 @@ def getCubeHostnames():
 
     # at mtc, assume cube license is available
     if fqdn.endswith("mtc.ca.gov"): return [ gethostname().lower() ]
-
-    f = open(r"Y:\COMMPATH\HostnamesWithCube.txt")
+    env = copy.copy(os.environ)
+    f = open(env['CUBE_HOST_FILE'])
     for line in f:
         if line[0] == "#": continue
         hostnames.append(line.split()[0])  # use the first token of non-comment lines
@@ -88,10 +88,9 @@ def export_cubenet_to_csvs(file, extra_link_vars=[], extra_node_vars=[],
                 sys.exit(2)
 
             env["MACHINES"] = CUBE_COMPUTER
-
-            cmd = r'y:\champ\util\bin\dispatch-one.bat "runtpp ' + script + '"'
-            print(cmd)
-            proc = subprocess.Popen( cmd, cwd = filedir, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
+
+            cmd = f'runtpp "{script}"'
+            proc = subprocess.Popen( cmd, cwd = filedir, stdout=subprocess.PIPE, stderr=subprocess.PIPE )#, env=env)
             for line in proc.stdout:
                 if type(line)==bytes: line = line.decode()  # convert to string, not byetes
                 line = line.strip('\r\n')

scripts/build_network.py

@@ -217,21 +217,21 @@ def writeRequirementsToScreen(REQUIREMENTS, req_type='prereq'):
     else:
         return None
 
-    print "Match type 2:   Perfect match    "
-    print "Match type 1:   Possible match   "
-    print "Match type 0:   No match         "
-    print "------------------------------   "
+    print("Match type 2:   Perfect match    ")
+    print("Match type 1:   Possible match   ")
+    print("Match type 0:   No match         ")
+    print("------------------------------   ")
 
     for net in REQUIREMENTS.keys():
         proj_name_max_width = 22
-        print "--------------------------------------------------------------------------------------------"
-        print "%s" % net.upper()
-        print "--------------------------------------------------------------------------------------------"
-        print "                       REQ    NET                          MATCH POSSIBLE               NET "
-        print "PROJECT                TYPE   TYPE  %-23sLEVEL %-23sTYPE" % (print_req.upper(), print_req.upper()+' MATCH')
-        print "---------------------- ------ ----- ---------------------- ----- ---------------------- ----"
+        print("--------------------------------------------------------------------------------------------")
+        print("%s" % net.upper())
+        print("--------------------------------------------------------------------------------------------")
+        print("                       REQ    NET                          MATCH POSSIBLE               NET ")
+        print("PROJECT                TYPE   TYPE  %-23sLEVEL %-23sTYPE" % (print_req.upper(), print_req.upper()+' MATCH'))
+        print("---------------------- ------ ----- ---------------------- ----- ---------------------- ----")
         if REQUIREMENTS[net].keys() == []:
-            print "NO %sS FOUND FOR %s NETWORK TYPE" % (print_req.upper(), net.upper())
+            print("NO %sS FOUND FOR %s NETWORK TYPE" % (print_req.upper(), net.upper()))
 
         for proj in REQUIREMENTS[net].keys():
             for req in REQUIREMENTS[net][proj].keys():
@@ -269,8 +269,8 @@ def writeRequirementsToScreen(REQUIREMENTS, req_type='prereq'):
                         else:
                             line_to_print = line_to_print + '\n' + line_part_one + "%-6s%-23s%-4s\n" %("NA","MISSING","NA")
                     i += 1
-                print line_to_print
-        print '\n'
+                print(line_to_print)
+        print('\n')
 
 def getProjectAttributes(project):
     # Start with TAG if not build mode, no kwargs
@@ -303,7 +303,7 @@ def getProjectAttributes(project):
     os.environ['CHAMP_NODE_NAMES'] = CHAMP_NODE_NAMES
 
     if len(args) < 1:
-        print USAGE
+        print(USAGE)
         sys.exit(2)        
     NETWORK_CONFIG  = args[0]
 
@@ -322,7 +322,7 @@ def getProjectAttributes(project):
         if o=="-c": CONFIG_WORD = a
 
     if BUILD_MODE not in [None,"test"]:
-        print USAGE
+        print(USAGE)
         sys.exit(2)
 
     if BUILD_MODE=="test":
@@ -334,25 +334,25 @@ def getProjectAttributes(project):
 
     # Verify mandatory fields are set
     if PROJECT==None:
-        print "PROJECT not set in %s" % NETWORK_CONFIG
+        print(f"PROJECT not set in {NETWORK_CONFIG}")
         sys.exit(2)
     if YEAR==None:
-        print "YEAR not set in %s" % NETWORK_CONFIG
+        print(f"YEAR not set in {NETWORK_CONFIG}")
         sys.exit(2)
     if SCENARIO==None:
-        print "SCENARIO not set in %s" % NETWORK_CONFIG
+        print(f"SCENARIO not set in {NETWORK_CONFIG}")
         sys.exit(2)
     if TAG==None:
-        print "TAG not set in %s" % NETWORK_CONFIG
+        print(f"TAG not set in {NETWORK_CONFIG}")
         sys.exit(2)
     if OUT_DIR==None:
-        print "OUT_DIR not set in %s" % NETWORK_CONFIG
+        print(f"OUT_DIR not set in {NETWORK_CONFIG}")
         sys.exit(2)
     if TRANSIT_CAPACITY_DIR==None:
-        print "TRANSIT_CAPACITY_DIR not set in %s" % NETWORK_CONFIG
+        print(f"TRANSIT_CAPACITY_DIR not set in {NETWORK_CONFIG}")
         sys.exit(2)
     if NETWORK_PROJECTS==None:
-        print "NETWORK_PROJECTS not set in %s" % NETWORK_CONFIG
+        print(f"NETWORK_PROJECTS not set in {NETWORK_CONFIG}")
         sys.exit(2)
 
     # Set up logging
@@ -464,7 +464,7 @@ def getProjectAttributes(project):
                 (prereqs, coreqs, conflicts) = networks[netmode].getReqs(networkdir=project_name, projectsubdir=tail, tag=tag,
                                                                          projtype=projType, tempdir=TEMP_SUBDIR)                
 
-            print "Checking projType... %s" % projType
+            print(f"Checking projType... {projType}") 
             if projType=='plan':
                 #Open specs file and get list of projects
                 specFile = os.path.join(TEMP_SUBDIR,NETWORK_PLAN_SUBDIR,'planSpecs.csv')
@@ -675,4 +675,4 @@ def getProjectAttributes(project):
 
     Wrangler.WranglerLogger.debug("Wrote transit report to %s" % transit_report_filename)
     Wrangler.WranglerLogger.debug("Successfully completed running %s" % os.path.abspath(__file__))
-    print "Remember to copy MissionLocalDelay.csv from the Muni_2011Oct dir!"
+    print("Remember to copy MissionLocalDelay.csv from the Muni_2011Oct dir!")

scripts/build_network_mtc_futures.py

@@ -201,9 +201,9 @@
             continue
 
         # Initialize output subdirectories up a level (not in scratch)
-        hwypath=os.path.join("..", SCENARIO, OUT_DIR.format(YEAR),HWY_SUBDIR)
+        hwypath=os.path.join("..", SCENARIO, OUT_DIR.format(YEAR),build_network_mtc.HWY_SUBDIR)
         if not os.path.exists(hwypath): os.makedirs(hwypath)
-        trnpath = os.path.join("..", SCENARIO, OUT_DIR.format(YEAR),TRN_SUBDIR)
+        trnpath = os.path.join("..", SCENARIO, OUT_DIR.format(YEAR),build_network_mtc.TRN_SUBDIR)
         if not os.path.exists(trnpath): os.makedirs(trnpath)
 
         networks['hwy'].write(path=hwypath,name=HWY_NET_NAME,suppressQuery=True,

scripts/net_spec_FBP_AL_021_South_Bay_Connect.py

@@ -0,0 +1,82 @@
+import os
+
+# MANDATORY. Set this to be the Project Name.
+# e.g. "RTP2021", "TIP2021", etc
+PROJECT = "PBA50-1"
+
+# MANDATORY. Set this to be the Scenario Name
+# e.g. "Base", "Baseline"
+SCENARIO = "FBP_AL_021_South_Bay_Connect"
+
+# MANDATORY. Set this to be the git tag for checking out network projects.
+TAG = "HEAD"
+
+# MANDATORY. Set this to the directory in which to write your outputs.
+# "hwy" and "trn" subdirectories will be created here.
+OUT_DIR = SCENARIO + "_network_{}"  # YEAR
+
+# MANDATORY.  Should be a dictionary with keys "hwy", "muni", "rail", "bus"
+# to a list of projects.  A project can either be a simple string, or it can be
+# a dictionary with with keys 'name', 'tag' (optional), and 'kwargs' (optional)
+# to specify a special tag or special keyword args for the projects apply() call.
+# For example:
+#     {'name':"Muni_TEP", 'kwargs':{'servicePlan':"'2012oct'"}}
+NETWORK_PROJECTS = collections.OrderedDict([
+    (2015, 
+        {'hwy':[
+            'PROJ_attributes' # adds PROJ attributes to NODE and LINK
+        ],
+        'trn':[]
+    }),  
+    (2020, {
+        'hwy':[
+            #{'name':'demo_project',        'kwargs':{'TEST':'BUNNIES'}}
+        ], 
+        'trn':[]
+    }),
+    (2025, {
+        'hwy':[
+        ],
+        'trn':[
+            #{'name':'demo_project',        'kwargs':{'FUTURE':"'BackToTheFuture'"}}
+        ]
+    }),
+    (2030, {
+        'hwy':[], 
+        'trn':[]
+    }),
+    (2035, {
+        'hwy':[],
+        'trn':[]
+    }),
+    (2040, {
+        'hwy':[], 
+        'trn':[]
+    }),
+    (2045, {
+        'hwy':[],
+        'trn':[]
+    }),
+    (2050, {
+        'hwy':["FBP_AL_021_South_Bay_Connect"],
+        'trn':["FBP_AL_021_South_Bay_Connect"]
+    })
+])
+
+# OPTIONAL. The default route network project directory is Y:\networks.  If
+# projects are stored in another directory, then use this variable to specify it.
+# For example: Y:\networks\projects
+# NETWORK_BASE_DIR = None
+# NETWORK_PROJECT_SUBDIR = None
+# NETWORK_SEED_SUBDIR = None
+# NETWORK_PLAN_SUBDIR = None
+
+# OPTIONAL. A list of project names which have been previously applied in the
+# PIVOT_DIR network that projects in this project might rely on.  For example
+# if DoyleDrive exists, then Muni_TEP gets applied differently so transit lines
+# run on the new Doyle Drive alignment
+APPLIED_PROJECTS = None
+
+# OPTIONAL.  A list of project names.  For test mode, these projects won't use
+# the TAG.  This is meant for developing a network project.
+TEST_PROJECTS = []