A/UX was a version of Unix Apple worked on from 1988 to 1995. It deserves a whole series of posts on this site, which I’ll get to eventually.
I was surprised to find several articles about it on Apple’s support site, including this one:
To run A/UX on the Macintosh IIfx, you must have A/UX 2.0 or any later version. Version 1.1 gave you this particular error message, because it doesn’t know how to use the IIfx DMA SCSI scheme. If you want to use A/UX on a IIfx, you must upgrade.
I really should pick up a Macintosh IIfx at some point. That thing was a badass machine in its day.