Some felt staff victimised them because of their religion and reported less interaction with warders, an inspection report added.
Criminal Justice Inspection Northern Ireland (CJI) said independent experts should investigate but praised other immensely encouraging work at the County Londonderry jail.
Chief inspector of criminal justice Brendan McGuigan said: “Poorer outcomes for Catholic prisoners remain in a number of key areas and it is our view that there needs to be a greater focus on the underlying reasons for these differences.
“It is our view the Northern Ireland Prison Service (NIPS) needs expert independent support to achieve this.”
Linfield v Portadown Sunday clash: Breakers of the Sabbath will answer to God
NI Weather: 45 pictures of how we enjoyed wall-to-wall sunshine
Tom Topping: Heartbroken daughter says ‘world is totally shattered’ after father’s shock death - funeral details released
Michelle O’Neill has kicked sand in the face of victims of terrorism
Belfast Pride 2022 organisers ‘may have broken law by seeking to bar transgender critics from parade’
A total of 16% of Catholics felt they had been victimised by staff because of their religion compared to 11% of Protestants.
While 11% felt victimised because of their nationality compared to 5% of Protestants.
The report said Catholic prisoners reported much less interaction with staff than Protestants.
Inspectors noted poorer outcomes for Catholics in a range of “important” areas, often where staff discretion was involved.
The report said: “In our survey Catholic prisoners were more negative than Protestants about respectful treatment by staff.
“While likely to be complex, the prison needed to do far more to understand the underlying reasons for this which was likely to require external expert assistance.
“There was promotion of some aspects of diversity, but a lack of a more proactive approach to all the protected groups covered by Northern Ireland law.”
The joint inspection was carried out by CJI and Her Majesty’s Inspectorate of Prisons in England and Wales (HMIP) with support from colleagues from the Regulation and Quality Improvement Authority (RQIA) and the Education and Training Inspectorate (ETI).
It found improvements in several key areas.
However, the chief inspectors were concerned by some inspection findings and called for further improvements to address ongoing issues.
They remained worried that an integrated drugs and alcohol strategy still did not exist at Magilligan Prison aimed at reducing supply and addressing the needs of prisoners with substance misuse or addiction problems.
Peter Clarke, Her Majesty’s Chief Inspector of Prisons in England and Wales, said: “Reducing the supply and use of illicit and illegal drugs is a major challenge, which requires a strategic approach.
“We recommend there should be a prison-wide drug and alcohol strategy with an associated action plan to address both supply reduction and support issues.”
The report noted some excellent and innovative support to disabled and older prisoners. The chaplaincy continued to provide a good service, including a varied programme of activities and good pastoral care.
When inspectors visited in 2014, they were concerned that nearly half the prison population was not participating in education, vocational training or work activities.
When they visited in June this year they found excellent progress had been made in this regard.
The Head of the Northern Ireland Prison Service, Ronnie Armour, said the report demonstrated the progress which has been made at Magilligan.
“By creating opportunities to improve skills, employability and self-confidence, those in our care are encouraged to address their offending behaviour and work towards playing a constructive role when they return to the community.
“Importantly, the report highlights that rehabilitation is truly at the heart of all we do at Magilligan Prison, with resettlement work identified as a significant strength.
“This, along with the strong work we are doing to support men in maintaining relationships with their children, families and friends, is helping to support people to change their offending behaviour and, hopefully, build safer communities.”
He said the report also highlights where Magilligan can continue to progress, in particular in supporting vulnerable prisoners and those with self-harm or mental health issues.
“As ever, we will continue to work to ensure that all those in our care have the support they need.”