Prereqs (pretty soft):
- Basic python knowledge, code editor set up to run scripts
- Very basic linux/unix knowledge
- Willing to learn new things/explore unfamiliar environments
Steps to access remote:
1. Generate an ssh for your device if you don’t have one → this will be used to
access the remote computer where all the App Inventor data files are stored
2. Ask one of the engineers (Evan or David) to add your public ssh to the remote machine
3. To access the data & run stuff, ssh into the remote with the command:
ssh datascience@[Link]
- Relevant files are in Documents
4. You can access the session people are working on via tmux:
tmux attach-session
5. To download or upload files, you can use ‘sftp’:
sftp datascience@[Link]
- the only commands you need to know are `ls`, `cd`, `get`, `put`
Information about the data & analysis tools:
Github link:
- [Link]
- [Link] is compressed CSV file of information about all projects
VSCode SSH GUI Extension (Optional)
It’s pretty inconvenient to edit files directly from the command line → it’s much
better to have a code editor. Here’s how I set up the GUI for editing remote files on
VSCode:
- Download the extension SSH FS
- Add a new configuration file
- Name it whatever you want (“app-inventor”)
- Host: [Link]
- Port: 22
- Root (optional): ~/Documents/data-tools-analysis
- Username: datascience
- Private key: the path to your private ssh key, typically of the format /Users/<your
username>/.ssh/<key id>
- New file mode: Check owner, Execute
- Press ‘Save’
- To access the remote, press the ‘Add as Workspace folder’ button in the configuration
Things I did:
- After a big refactor, I organized all my stuff into a folder called ‘analysis’, into which i
have a ‘scripts’ folder and an ‘out’ folder (for output csv’s and stuff)
AIA TOOLS:
[Link]
[Link]
>> import os
>> os. (find when last modified)
- build a map of block id to when block was instantiated, used that to ‘determine’ which
project is templated off which block
- theory, if you make a copy use ‘save as’, blocks ID’s are copies
- ‘backpack’
- need template files
>>> from aiatools import AIAFile
>>> project = AIAFile(‘Users/…something’)
>>> [Link]()
[Block(‘P9D2qDc…[block id]’, ‘component_event’), Block(‘id’,
‘component_all_component_block’)...
)]
>>> block_ids = set([[Link] for block in [Link]()])
>>> block_ids
Hi Robert & Evan,
I just had some questions regarding the aiatools and stuff. From what I've seen so
far, it looks like one can only open one AIA file at a time:
<[Link]>
And from then on, I can call on various functions, of which I can extract the block id’s.
Is there a way, perhaps using the compressed output file, I can open up a bunch of projects at
once, perhaps without even downloading them---is there perhaps a file location or something
embedded in the output csv? It would also probably be really helpful to have a local directory of
downloaded App Inventor tutorial files. I’m just a bit confused at how to approach getting the
block id’s of each project.
Also, regarding the normalized block usage analysis, it would be really great if I could get
information on when each block was released, though no rush at all. Thank you so much!
Sincerely,
Abrianna
Hi Abrianna,
The general flow should go something like this (or at least, this is how I've done my analyses for a
number of write-ups and presentations):
First, get access to the AIAs of interest. For the data science analysis, the files are in
~/Documents/output_aias.
For the tutorials, I've converted them to AIA files and you can find them in ~/Documents/tutorial_aias.
As a rough sketch:
1. Walk the directory tree where the tutorial AIAs are stored using [Link]. Use with
[Link](...) as project: to open a context.
2. For each project, you can get the set of block IDs with the following snippet:
set([Link]().select(lambda x: [Link]))
3. Run this first to get the list of IDs for each of the tutorials, then you can run it over the set of user
projects and compare them. I think the best way to do this is to first combine all of the tutorial block ids
into a single set that is the union of all block ids--let's call it tutorial_block_ids--and then use
tutorial_blocks_ids.isdisjoint(project_block_ids) to test for disjointness, i.e., if
the project is not based on a tutorial this should return True.
As an extra exercise, if the tests passes, you could then check the individual tutorials to see which one
intersects the individual project to determine the project's origin.
I can chat more over Zoom if it would be helpful.
Regards,2
Evan
[Link]
- unzip with unxz [Link]
- contains about 550k rows or tuples:
- Project ID, User Hash, Creation Date, Modification Date, Component
Components covered:
* BluetoothLE
* Microbit_Uart_Simple
* ChartData2D
* Chart
* ChartData2D
* Trendline
* AnomalyDetection
* Spreadsheet
* ChatBot
* ImageBot
* LookExtension
* PersonalImageClassifier
* PersonalAudioClassifier
* PosenetExtension
* FaceExtension
- Find tuple of project rg <project_id> [Link]
- query script: python3 [Link] --extract <Component1> <(opt.) Component2> …