Tru64 AdvFS porting to NetBSD - 2. status 2014-10-11

[Kamil Rytarowski]

Hello,

This is the second status [1] of porting AdvFS to NetBSD.

Thank you for your motivation support, including mails from outside the NetBSD world.

1. What is done
- Moved AdvFS files from src/sys/fs/msfs to src/sys/external/gpl2/msfs and updated the build machinery
- Picked missing dyn_hashtable functionality from the HP/UX port of AdvFS
- Designed new debugging & tracing system, with changeable levels (none, fatal asserts, debug asserts, extensive checks) it's intended to replace the existing fine-grained debugging that is placed in the original work almost everywhere and it's impractically difficult to port 1:1, as it's utilizing Tru64-specific features -- the HP/UX port went with similar path; most debugging code (most notably related to locking) will be gone
- Stopped using indent(1) as it introduces a lot of harm because of extensive usage of macros (missing semicolons etc..)
- Overall: cleaned and squashed 377 proof-of-concept (aka throw-away) commits [2] into 115 cleaned revisions (aka throw-away later) [3]
2. What is in progress
- Adapting locking code, with verification of the right path with the HP/UX port
- Adapting debugging for new design, removing unneeded code-complication and Tru64-specific debug solutions
- Converting macros, used as in-lined functions with side-effects, to functions
- Removing alternative compilation paths (exception for _KERNEL in general and MSFS_DEBUG in msfs/ms_assert.h)
- Other compatibility patches for a modern compiler and NetBSD
3. Issues
- Missing subsystems' details from Tru64, still no idea about definitions of functions from overlap.h, missing quota's code (but not looked at it closely)
4. Next steps
- Virtual Memory porting
5. Pushed to NetBSD
- Proposed new <sys/clock.h> patch [4] still pending :-( and blocking further reduction of MD code..

Help and motivation support is still appreciated.

[1] Previous status at http://mail-index.netbsd.org/tech-kern/2014/09/17/msg017684.html
[2] https://github.com/krytarowski/netbsd-current-src-sys/tree/advfs_2014-09-27_old
[3] https://github.com/krytarowski/netbsd-current-src-sys/tree/advfs
[4] http://mail-index.netbsd.org/netbsd-bugs/2014/10/08/msg038523.html & http://mail-index.netbsd.org/netbsd-bugs/2014/10/09/msg038531.html

[Thor Lancelot Simon]

On Sat, Oct 11, 2014 at 10:52:21PM +0200, Kamil Rytarowski wrote:

> - Designed new debugging & tracing system, with changeable levels (none, fatal asserts, debug asserts, extensive checks) it's intended to replace the existing fine-grained debugging that is placed in the original work almost everywhere and it's impractically difficult to port 1:1, as it's utilizing Tru64-specific features -- the HP/UX port went with similar path; most debugging code (most notably related to locking) will be gone

Are you sure a new dtrace provider would not be the way to go?

--
Thor Lancelot Simon ***@panix.com
"From the tooth paste you use in the morning to the salt on your evening meal,
it's easy to take for granted the many products brought to us with explosives."
- Institute of Manufacturers of Explosives, "Explosives Make It Possible"

[Kamil Rytarowski]

Hello,

> Are you sure a new dtrace provider would not be the way to go?

Yes. I believe that internal performance tracing and tracking locks is misdesign (also not very portable). I'm not familiar with DTrace, more with Linux tools (perf, valgrind) and my general plan was to debug it with librump in Linux.

Of course the first milestone is not performance optimization but getting it well integrated with NetBSD and stable. That's why I took Tru64 long-term tested release, not HP/UX that was never tested in production.

Some checks will be needed anyway, like function assumes that a lock is acquired in a certain function - with asserts such bugs will be narrowed-down quickly.

[Justin Cormack]

On Sun, Oct 12, 2014 at 11:06 AM, Kamil Rytarowski <***@gmx.com> wrote:
> Hello,
>
>> Are you sure a new dtrace provider would not be the way to go?
>
> Yes. I believe that internal performance tracing and tracking locks is misdesign (also not very portable). I'm not familiar with DTrace, more with Linux tools (perf, valgrind) and my general plan was to debug it with librump in Linux.
>
> Of course the first milestone is not performance optimization but getting it well integrated with NetBSD and stable. That's why I took Tru64 long-term tested release, not HP/UX that was never tested in production.
>
> Some checks will be needed anyway, like function assumes that a lock is acquired in a certain function - with asserts such bugs will be narrowed-down quickly.

The Linux LTTng performance tool is basically compatible with NetBSD
dtrace hooks, ie you can use them on rump kernel code in Linux
userspace.

See https://github.com/rumpkernel/wiki/wiki/Howto%3A-Profiling-the-TCP-IP-stack-with-LTTng

And this way you end up with dtrace support under NetBSD too.

Justin

[Kamil Rytarowski]

Justin Cormack wrote
> The Linux LTTng performance tool is basically compatible with NetBSD
> dtrace hooks, ie you can use them on rump kernel code in Linux
> userspace.
>
> See https://github.com/rumpkernel/wiki/wiki/Howto%3A-Profiling-the-TCP-IP-stack-with-LTTng

Thank you!

Actually some tracing admin-oriented is desired, please see advfsstat(8) [1]. A developer-oriented tracing IMO shall to be done with external tools, not from inherited code. For the latter LTTng looks promising!

[1] http://h50146.www5.hp.com/products/software/oe/tru64unix/manual/v51a_ref/HTML/MAN/MAN8/0351____.HTM