This should fix #3017

Describe the bug The method getServerRepository(String host, int port, String repositoryName, boolean createIfNeeded) of class ghidra.base.project.GhidraProject is ignoring the parameter createIfNeeded and always creates the repository if non existent.

Expected behavior If createIfNeeded is false, I would expect no repository to be created if it doesn't exist.

DAT_143bcce50 = CONCAT412(uStack12,CONCAT48(DAT_1430fb9b8,(ulonglong)DAT_1430fb9a4));

How to disable CONCAT and export readable code instead of CONCAT????

Bumps url-parse from 1.4.7 to 1.5.1. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/unshiftio/url-parse/commit/eb6d9f51e395b7e47bf2594e457d541db21c713b"><code>eb6d9f5</code></a> [dist] 1.5.1</li> <li><a href="https://github.com/unshiftio/url-parse/commit/750d8e8a9d45dbce9ff09759f0fe4564cdd47d74"><code>750d8e8</code></a> [fix] Fixes relative path resolving <a href="https://github-redirect.dependabot.com/unshiftio/url-parse/issues/199">#199</a> <a href="https://github-redirect.dependabot.com/unshiftio/url-parse/issues/200">#200</a> (<a href="https://github-redirect.dependabot.com/unshiftio/url-parse/issues/201">#201</a>)</li> <li><a href="https://github.com/unshiftio/url-parse/commit/3ac777474ba5dc48a7e33771cbb311fc6f69bef8"><code>3ac7774</code></a> [test] Make test consistent for browser testing</li> <li><a href="https://github.com/unshiftio/url-parse/commit/267a0c6f7ef1a58271be61611c5103daace602c9"><code>267a0c6</code></a> [dist] 1.5.0</li> <li><a href="https://github.com/unshiftio/url-parse/commit/d1e7e8822f26e8a49794b757123b51386325b2b0"><code>d1e7e88</code></a> [security] More backslash fixes (<a href="https://github-redirect.dependabot.com/unshiftio/url-parse/issues/197">#197</a>)</li> <li><a href="https://github.com/unshiftio/url-parse/commit/d99bf4cf259b7378c855f786edc253e70405ffdc"><code>d99bf4c</code></a> [ignore] Remove npm-debug.log from .gitignore</li> <li><a href="https://github.com/unshiftio/url-parse/commit/422c8b5e4cac6a79cd35b4e86731476dcbeec7e4"><code>422c8b5</code></a> [pkg] Replace nyc with c8</li> <li><a href="https://github.com/unshiftio/url-parse/commit/933809d630c7b21399b4e5df59fccccd80033b21"><code>933809d</code></a> [pkg] Move coveralls to dev dependencies</li> <li><a href="https://github.com/unshiftio/url-parse/commit/190b2168035899a2a88f2dc2625963bf7e2f338f"><code>190b216</code></a> [pkg] Add .npmrc</li> <li><a href="https://github.com/unshiftio/url-parse/commit/ce3783f4ea25753cfa36376769c14e4e2fe6ea80"><code>ce3783f</code></a> [test] Do not test on all available versions of Edge and Safari</li> <li>Additional commits viewable in <a href="https://github.com/unshiftio/url-parse/compare/1.4.7...1.5.1">compare view</a></li> </ul> </details> <br />

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

<details> <summary>Dependabot commands and options</summary> <br />

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
• @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
• @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
• @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
• @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

</details>

Setting the values of 2 individual registers that share a double-wide overlapping register causes the decompiler to display memory references as offsets from a variable instead of showing the actual address.

I suspect this problem stems from the fact that the V850 compiler spec has the 2 individual registers (gp and tp) marked as unaffected, but not the double-wide register (r4r5).

gp and tp were set individually by selecting the entire binary and using the set register context menu.

You can see in the function header below that Ghidra has collapsed the gp and tp registers into the r4r5 register.

At addresses 0x371E and 0x3722 we see the source addresses of our two load instructions being displayed as memory references. However, in line 22 of the decompiled view, we see the source addresses of the load instructions displayed as offsets from two variables, instead of the actual calculated values from the disassembly view.

The image below shows the behavior I expect from the decompiled view. Note how the two loads in line 20 are displayed as addresses and not offsets from variables.

Environment:

• OS: Windows 10 Build 19041
• Java Version: 14.0.2
• Ghidra Version: 9.2.3
• Ghidra Origin: official ghidra-sre.org distro

Additional context I have found 2 workarounds for this problem specific to V850.

• The simplest is to add the r4r5 register to the unaffected list of the compiler spec. That is how I got the expected behavior screenshot.
• The worse solution is to remove the overlapping register (r4r5).

Minor correction to gradle building instructions.

I'll take a look at it as time allows. There are different categories of memory leaks that warrant more or less concern when it comes to UI components.

If you have the aux registers

define register offset=0x1000 size=4 [ aux000 ...
                                              ...
                                              ... auxfff ];

in the pspec file you can have (copying from AARCH64). It would then look similar to you current pcodeop setup in the decompiller in that it looks like function call, but it would be able to move the data around.

<volatile outputop="AUXWrite" inputop="AUXRead">
    <range space="register" first="0x1000" last="0x4fff"/>
</volatile>

I know pcodetest.py currently has -nodefaultlibs, but wonder if you also need -nostdlib (if that doesn't make it worse), ideally gcc shouldn't link to libm type things, but I guess that assumes the non-optimized version gcc can generate doesn't also use norm

Describe the bug After window that displayed table is closed, table model data is not garbage collected.

To Reproduce Steps to reproduce the behavior:

1. Launch tool in Search - For Address Tables...
2. A window pop up.
3. Click Search.
4. Dismiss the window.
5. Launch visualvm and attach ghidra
6. Take heap snapshot
7. Click panel that is with title of instances by size.
8. HashMap$Node[] is placed at first. Expanding subtree reveals that table control is still referencing model.

Attachments A patch file is attached. 0001-210508.patch.txt

Environment (please complete the following information):

• Ghidra Version: [e.g. 9.2.2]

Bumps ua-parser-js from 0.7.21 to 0.7.28. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/faisalman/ua-parser-js/commit/1d3c98a10c23915046a362c4e3b3b503fb40d611"><code>1d3c98a</code></a> Revert breaking fix <a href="https://github-redirect.dependabot.com/faisalman/ua-parser-js/issues/279">#279</a> and release as 0.7.28</li> <li><a href="https://github.com/faisalman/ua-parser-js/commit/535f11bd2403910f29dabe6f90adb014ad016747"><code>535f11b</code></a> Delete redundant code</li> <li><a href="https://github.com/faisalman/ua-parser-js/commit/642c0399e831e27c5c86c3b7afee02e876250d01"><code>642c039</code></a> Fix <a href="https://github-redirect.dependabot.com/faisalman/ua-parser-js/issues/492">#492</a> LG TV WebOS detection</li> <li><a href="https://github.com/faisalman/ua-parser-js/commit/3edacddb2474429c55fa39d1e6222d50bbf9266f"><code>3edacdd</code></a> Merge branch 'master' into develop</li> <li><a href="https://github.com/faisalman/ua-parser-js/commit/acc0b91ff5defa2ca9a722874e27277879292907"><code>acc0b91</code></a> Update contributor list</li> <li><a href="https://github.com/faisalman/ua-parser-js/commit/f726dcd1ae4fea51e99b8ca574a1be51fbd70e1a"><code>f726dcd</code></a> Merge branch 'master' into develop</li> <li><a href="https://github.com/faisalman/ua-parser-js/commit/383ca587ef9b8daffcf652ac39fc9b8f3708572e"><code>383ca58</code></a> More test for tablet devices</li> <li><a href="https://github.com/faisalman/ua-parser-js/commit/7c8aa435b26cb14537423cd5fe4ce077e0661db2"><code>7c8aa43</code></a> Minor rearrangement</li> <li><a href="https://github.com/faisalman/ua-parser-js/commit/09aa9105dc370ded9275f70eae1f4eb67394966c"><code>09aa910</code></a> Add new device & browser: Tesla</li> <li><a href="https://github.com/faisalman/ua-parser-js/commit/557cc2198d88068892eec6b61f2b2f4fe6e96314"><code>557cc21</code></a> More test for latest phones with unique form factor (fold/flip/qwerty/swivel)</li> <li>Additional commits viewable in <a href="https://github.com/faisalman/ua-parser-js/compare/0.7.21...0.7.28">compare view</a></li> </ul> </details> <br />

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

<details> <summary>Dependabot commands and options</summary> <br />

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
• @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
• @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
• @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
• @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

</details>

@mumbel For the AUX registers, I began a PoC and encountered issues because the decompiler did not consider the register as "volatile".

For example when interacting with the MMU (like the code I share in a previous comment, https://github.com/NationalSecurityAgency/ghidra/pull/3006#issuecomment-834200211), when writing a value to AUX register 0x408, the value of AUX register 0x407 changes based on the values in registers 0x405 and 0x406. And when writing another value to 0x408 (such as 2 to issue a TLBRead command, cf. http://me.bios.io/images/7/73/ARC700_MemoryManagementUnit_Reference.pdf for details), the values in registers 0x405 and 0x406 change depending on the one in register 0x407. This behavior is like hardware registers in embedded devices. Is there a way to declare registers as having such a behavior? (Currently, if I use registers in SLEIGH, the decompiler feels free to reorder the reads and stores).

For the pcodetests, I am hitting an issue: in the implementation of integer division (in gcc's function __udivmodsi4), the instruction norm is used. It is defined as a pcodeop so the emulation does not work. This instruction looks like "Count Leading Zeros" instruction in other architectures (even though it is not exactly a CLZ): Aarch64's clz, MIPS64's dclz, RISC V's clz... When looking at SLEIGH files for these architectures, I always see that the CLZ instruction is defined with a pcodeop. Does this mean that I need to find a way to implement the emulation of norm for example in Java, to make the emulator work? Or is there some flag which states that the division cannot be emulated in pcodetests? How would you like this issue to be solved?

@GhidorahRex in order to make the pcodetests run, I also needed to modify the file build.gradle for ARCompact processor. I copied what I saw in other processors and added the Base dependency, which is required by pcodetests:

diff --git a/Ghidra/Processors/ARCompact/build.gradle b/Ghidra/Processors/ARCompact/build.gradle
index 5f211ea089fa..f2234ca0e6e9 100644
--- a/Ghidra/Processors/ARCompact/build.gradle
+++ b/Ghidra/Processors/ARCompact/build.gradle
@@ -14,6 +14,13 @@
  * limitations under the License.
  */
 apply from: "$rootProject.projectDir/gradle/distributableGhidraModule.gradle"
+apply from: "$rootProject.projectDir/gradle/javaProject.gradle"
+apply from: "$rootProject.projectDir/gradle/jacocoProject.gradle"
+apply from: "$rootProject.projectDir/gradle/javaTestProject.gradle"
 apply from: "$rootProject.projectDir/gradle/processorProject.gradle"
 apply plugin: 'eclipse'
 eclipse.project.name = 'Processors ARCompact'
+
+dependencies {
+       compile project(':Base')
+}

Is such a process documented somewhere?

Bumps ua-parser-js from 0.7.21 to 0.7.28. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/faisalman/ua-parser-js/commit/1d3c98a10c23915046a362c4e3b3b503fb40d611"><code>1d3c98a</code></a> Revert breaking fix <a href="https://github-redirect.dependabot.com/faisalman/ua-parser-js/issues/279">#279</a> and release as 0.7.28</li> <li><a href="https://github.com/faisalman/ua-parser-js/commit/535f11bd2403910f29dabe6f90adb014ad016747"><code>535f11b</code></a> Delete redundant code</li> <li><a href="https://github.com/faisalman/ua-parser-js/commit/642c0399e831e27c5c86c3b7afee02e876250d01"><code>642c039</code></a> Fix <a href="https://github-redirect.dependabot.com/faisalman/ua-parser-js/issues/492">#492</a> LG TV WebOS detection</li> <li><a href="https://github.com/faisalman/ua-parser-js/commit/3edacddb2474429c55fa39d1e6222d50bbf9266f"><code>3edacdd</code></a> Merge branch 'master' into develop</li> <li><a href="https://github.com/faisalman/ua-parser-js/commit/acc0b91ff5defa2ca9a722874e27277879292907"><code>acc0b91</code></a> Update contributor list</li> <li><a href="https://github.com/faisalman/ua-parser-js/commit/f726dcd1ae4fea51e99b8ca574a1be51fbd70e1a"><code>f726dcd</code></a> Merge branch 'master' into develop</li> <li><a href="https://github.com/faisalman/ua-parser-js/commit/383ca587ef9b8daffcf652ac39fc9b8f3708572e"><code>383ca58</code></a> More test for tablet devices</li> <li><a href="https://github.com/faisalman/ua-parser-js/commit/7c8aa435b26cb14537423cd5fe4ce077e0661db2"><code>7c8aa43</code></a> Minor rearrangement</li> <li><a href="https://github.com/faisalman/ua-parser-js/commit/09aa9105dc370ded9275f70eae1f4eb67394966c"><code>09aa910</code></a> Add new device & browser: Tesla</li> <li><a href="https://github.com/faisalman/ua-parser-js/commit/557cc2198d88068892eec6b61f2b2f4fe6e96314"><code>557cc21</code></a> More test for latest phones with unique form factor (fold/flip/qwerty/swivel)</li> <li>Additional commits viewable in <a href="https://github.com/faisalman/ua-parser-js/compare/0.7.21...0.7.28">compare view</a></li> </ul> </details> <br />

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

<details> <summary>Dependabot commands and options</summary> <br />

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
• @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
• @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
• @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
• @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

</details>

Bumps url-parse from 1.4.7 to 1.5.1. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/unshiftio/url-parse/commit/eb6d9f51e395b7e47bf2594e457d541db21c713b"><code>eb6d9f5</code></a> [dist] 1.5.1</li> <li><a href="https://github.com/unshiftio/url-parse/commit/750d8e8a9d45dbce9ff09759f0fe4564cdd47d74"><code>750d8e8</code></a> [fix] Fixes relative path resolving <a href="https://github-redirect.dependabot.com/unshiftio/url-parse/issues/199">#199</a> <a href="https://github-redirect.dependabot.com/unshiftio/url-parse/issues/200">#200</a> (<a href="https://github-redirect.dependabot.com/unshiftio/url-parse/issues/201">#201</a>)</li> <li><a href="https://github.com/unshiftio/url-parse/commit/3ac777474ba5dc48a7e33771cbb311fc6f69bef8"><code>3ac7774</code></a> [test] Make test consistent for browser testing</li> <li><a href="https://github.com/unshiftio/url-parse/commit/267a0c6f7ef1a58271be61611c5103daace602c9"><code>267a0c6</code></a> [dist] 1.5.0</li> <li><a href="https://github.com/unshiftio/url-parse/commit/d1e7e8822f26e8a49794b757123b51386325b2b0"><code>d1e7e88</code></a> [security] More backslash fixes (<a href="https://github-redirect.dependabot.com/unshiftio/url-parse/issues/197">#197</a>)</li> <li><a href="https://github.com/unshiftio/url-parse/commit/d99bf4cf259b7378c855f786edc253e70405ffdc"><code>d99bf4c</code></a> [ignore] Remove npm-debug.log from .gitignore</li> <li><a href="https://github.com/unshiftio/url-parse/commit/422c8b5e4cac6a79cd35b4e86731476dcbeec7e4"><code>422c8b5</code></a> [pkg] Replace nyc with c8</li> <li><a href="https://github.com/unshiftio/url-parse/commit/933809d630c7b21399b4e5df59fccccd80033b21"><code>933809d</code></a> [pkg] Move coveralls to dev dependencies</li> <li><a href="https://github.com/unshiftio/url-parse/commit/190b2168035899a2a88f2dc2625963bf7e2f338f"><code>190b216</code></a> [pkg] Add .npmrc</li> <li><a href="https://github.com/unshiftio/url-parse/commit/ce3783f4ea25753cfa36376769c14e4e2fe6ea80"><code>ce3783f</code></a> [test] Do not test on all available versions of Edge and Safari</li> <li>Additional commits viewable in <a href="https://github.com/unshiftio/url-parse/compare/1.4.7...1.5.1">compare view</a></li> </ul> </details> <br />

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

<details> <summary>Dependabot commands and options</summary> <br />

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
• @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
• @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
• @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
• @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

</details>

Reported via Play Dashboard;

java.lang.NullPointerException: 
  at com.aletheiaware.perspective.android.ui.MainActivity.onWindowFocusChanged (Unknown Source:15)

Occurrences/Devices;

• 7: Google Pixel 4a (5G)
• 5: Samsung Galaxy Tab S6 Lite
• 4: Samsung Galaxy Tab A with S Pen

We have a project consisting of multiple versions of the same basic program. These programs share the same DataTypes (in particular, structure definitions), so we have a common DataType archive to pull from. However, a persistent problem we see is illustrated by this example:

Consider messages passed between tasks within the program. Different versions of the program may have different definitions of particular message numbers, so the enum that defines the message number is unique for each program within the project. But, the message header itself is common between all versions, except that they use different enums for the message number. But, the DataType archive has to contain a single message number enum in the archive. Furthermore, message queues point to the message header, so they want to be unique based on which enum is in the header, etc.

So, what we want to do is to have a "generic" enum in the DataType archive and the specific values in the enum get used from the program datatype manager. At the moment, we use ushort in the message header, but this disables the very useful enum expansion that ghidra is capable of. And sometimes this variation is actually slightly different contents of structures, which is more problematic to work around.

So, the question is, are we overlooking a better way to deal with this problem?

Sorry, was very poorly worded. This may just be ARC inexperience, but I just meant the difference between:

• lr r0,[status32] -> you know its an aux op on status32
• lr r0,[aux407] -> you know its an aux op on aux register 0x407
• lr r0,[0x407] -> you see 0x407, is that an immediate and address, oh its instruction X, that's an aux register

By adding this as a register instead of an immediate value (which your display could remain immediate, but the implementation could use the registers), you could eliminate more pcodeops. Which are kind of logical breaks, w/o additional code elsewhere, it is a call where it doesn't know what's going on in the op or what going to be returned, that may not matter at all for ARC.

The next step is to implement the java to run the tests. https://github.com/NationalSecurityAgency/ghidra/tree/master/Ghidra/Processors/PowerPC/src/test.processors/java/ghidra/test/processors for example is the PowerPC. You would implement two classes ARCompact_O0_EmulatorTest and ARCompact_O3_EmulatorTest. Then inside of eclipse, you can right click and run unit tests on the class. It will look for those .out in Ghidra/Processors/ARCompact/pcodetests/ I think, so just copy the 2 files there. From there it should start unit tests by first creating the project for the .out and making a cached file. Then it will run some unit tests with a bunch of output with any failures reported at the end. You can narrow down and run individual tests to get asm printed insn by insn (adjust your output/console length) to narrow down on what is failing, there will be assert function calls/strings that you can use in the output to find where to work back from.

There are a few annoyances, like drag and drop is the only way to share, there is no context menu entry for "share with archive", so I have to drag an entry across 5 screens of datatypes to get to the library, which if it's expanded is easy to skip over the library itself...

Just between the two of us, I feel like our Data Type management never really matured.

(we also have a lot of home grown tools)

This really is the expectation. Obviously for user-specific needs, but also to address the tool's deficiencies, we assume our clients write into Ghidra the functionality that they need. The plugin and extension points features are designed to this end. Of course, being open source ultimately allows clients to make Ghidra whatever they need when the extendibility is not enough.

We have to walk a fine line when deciding which user-made features should be pulled back into the tool. Whatever we decide to put into the tool has to be useful and generic enough for us to easily maintain. As you are pointing out in this ticket, I think the overall Data Type workflow needs quite a bit of work. This is something that we should fix. Admittedly though, this is a hard problem. You have also pointed out that long-term management of RE'd libraries is something that has not gotten enough attention. This is something that likely would require more resources and collaboration for us to improve. Perhaps we need more great developers working on the Ghidra team... :dragon:

Yes, at the moment we have a two-layer hierarchy of data-type management. There is a top-layer archive with some basic types that seem pretty invariant. Then, as we see different families of binaries we break off datatypes into these family-specific archives. This is "manageable" but is a rather troublesome manual process. There are a few annoyances, like drag and drop is the only way to share, there is no context menu entry for "share with archive", so I have to drag an entry across 5 screens of datatypes to get to the library, which if it's expanded is easy to skip over the library itself...

And we do some things like using ushort for the message number in the structure of a generic message with a comment that this is the msgType because the enum is specific to that particular program (the issue above). This means that the message structure is sharable without causing conflicts if it had the actual enum datatype in the structure, but it also means that ghidra doesn't understand the data as well as it could.

BTW, we currently have ~30 binaries that are all related, and it is a struggle to manage this. Ghidra does a lot more than anything else I have ever seen. Things like bsim help (we also have a lot of home grown tools) and version tracker markup is the only thing that I've seen that helps with porting volatile data from one program to another. But it's a difficult problem which is still just starting to be solved.

Does that make any sense at all? I want to share structs and signatures that refer to msgType, but have a unique msgType for each binary (since each one has a different numbering).

I think this makes sense. You'd like a way to have certain types get replaced with program-specific versions when applied to a particular program. It seems like we'd need to have a new type, something like a Placeholder Type, that Ghidra knows must be resolved with a program-specific type. If that type does not exist, then the UI would flag that somehow so that you'd know to create that program's placeholder type before you can start using the containing data type.

but I want to share everything that only references struct foo between all of the different binaries.

Are you familiar with the notion of Data Type Synchronizing in Ghidra? This allows you to add types to a program archive from various other data type libraries. Then, as the types change, Ghidra shows that the source archive and program archive have diverged. There are actions to resynchronize the types in the archives. This is clearly not what you are asking for in this feature, but it does help somewhat with managing archives as types change.

@niooss-ledger this looks like a great start. I'll take a look when I get some time and see what additional work needs to be done. I'm grateful for the pcodetests as well. Those are always valuable. They provide support in two ways - the first is providing a static set of binaries so we can see that the control flow instructions are working correctly, and spot opcodes that are not disassembling.

The second is by running them with emulation. To do that, you need to define some java test libraries. Lots of our other processors have test.processor directories with the java code defining the emulation tests if you want some examples.

1. AUX registers

For the AUX registers, I think there are several difficulties in your approach. In order to make thinks clearer, here are some code examples.

In some firmware, the following code is used to interact with the MMU:

        c0089f94 ab 20 50 01                  sr         r0,[0x405]
        c0089f98 ab 21 90 01                  sr         r1,[0x406]
        c0089f9c ab 26 10 72 00 00 04 00      sr         0x4,[0x408]
        c0089fa4 aa 20 d0 01                  lr         r0,[0x407]                

The decompiler output (when using this Pull Request) is currenly:

  set_AUXREG(0x405,param_1); // Write a virtual address to TLB Page Descriptor register 0
  set_AUXREG(0x406,param_2); // Write a physical address to TLB Page Descriptor register 1
  set_AUXREG(0x408,4); // Write "TLBProbe" to TLB Command register
  uVar1 = get_AUXREG(0x407); // Read result from TLB Index register

In this case, it would be nice to have register names. The instructions are using 12 bits to define the immediate values, so there would be 4096 auxiliary registers defined in the SLEIGH specification.

Nevertheless some code use "dynamic" numbers to access auxiliary registers. For example:

        00021cca 38 61                   add_s      r0,r1,r1
        00021ccc 14 78                   add1_s     r0,r0,r0
        00021cce c7 70 80 00 8c d1       add_s      r0,r0,0x80d18c
        00021cd4 40 90                   ldw_s      r2,[r0=>DAT_0080d18c]
        00021cd6 01 e1                   add_s      r1,0x1
        00021cd8 2a 23 80 00             lr         r3,[r2]

Which is decompiled as:

    // uVar1 is r1
    uVar2 = get_AUXREG((uint)(ushort)(&DAT_0080d18c)[uVar1 * 3]);
      // Save the value of some AUX registers defined in a table in DAT_0080d18c

In this case, using hard-coded registers in the SLEIGH specification would not change anything.

With this in mind, I did not understand your comment:

Even if you didn't rename or even name any specific ones at all, won't instructions still be trying to use these registers, leaving undefined registers seems like disassembly information may suffer, analysis could fail or the all the dots don't get connected.

Currently, instructions using AUX registers are always decoded, using numbers instead of nice-looking names. Could you please explain why the disassembly information may suffer, the analysis could fail or the all the dots don't get connected?

5. pcodetest

To run pcodetests, I used a prebuilt toolchain from https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/releases/tag/arc-2020.09-release, created some symlinks (like ln -s arc-linux-gcc gcc) and modified Ghidra/Extensions/SleighDevTools/pcodetest/pcode_defs.py. With this, I successfully built two files named ARCompact_GCC_O0_pcodetest.out and ARCompact_GCC_O3_pcodetest.out.

When I open these files in Ghidra, they are correctly analyzed (and I get a warning about DWARF registers not supported, because until recently I only analyzed firmware with no debug information; I will work on adding some basic DWARF support too).

Is there some documentation about how the ..._pcodetest.out files can be used to find bugs? https://github.com/NationalSecurityAgency/ghidra/tree/master/Ghidra/Extensions/SleighDevTools/pcodetest seems to only describe the build process, and I might have missed some other places where the "next step" is documented..

This also seems to influence Rust decompiling, where a "and rsp, X" is used to align the stack. Patch out the and rsp, X gives the correct result. I will do more debug and source reading to see why this happens..

Hello. I am running WSL on Windows 10 and have connected to gdb-multiarch via GDB over SSH. I connected to my 2DS via Luma3DS via GDB and the inferior 1 - process 42000 - <null> shows up. I right clicked this and press Record. And it gives me the window shown below, when what I want is for it to populate the Threads window. The Threads window stays empty and so does everything else besides the Objects list.

Why would the Record button show this instead of showing me the Thread list?

Sorry if I am not too clear.