Born in neighbouring Democratic Republic of Congo in 1947, he was known as “the Doctor” because he was seen as a defender of the people.
His songs often criticised Tanzania’s elite and at the height of his popularity the government tried to expel him on immigration grounds.
But as his international fame grew, he was granted Tanzanian nationality.
His mix of Congolese “soukous” music and Swahili rhythms made him popular across East Africa.
He was also a regular performer at the Womad music festival in the UK until he fell ill 10 years ago and was paralysed.
Despite his ill-health he had toured in Tanzania until recently, mainly performing gospel music.
The BBC’s Hassan Mhelela in Dar es Salaam says all radio stations in the country are playing his music and fans are phoning in with tributes.
By the time of his death, even President Jakaya Kikwete was a fan, recently visiting him in hospital.
Ongala often wrote about death — one of his most well-known tracks is Kifo, about the mercilessness of death, saying no matter how rich a person, bribery cannot postpone it.
The singer and guitarist moved to Tanzania in 1977 and joined the Orchestra Super Makassy, until he formed his own group Super Matimila.
He said his music was intended not only to make people dance but also to think.
“I am successful in Tanzania because I write songs about serious topics,” his music label, Real World Records, quoted him as saying.
In 1990, he faced opposition to his song Mambo Kwa Socks (Things With Socks) — a reference to condoms — in which he made a plea to young men to practise safe sex.
Radio Tanzania has refused to play it, but live shows and black market tapes have ensured that his message has spread.
There is even a suburb of Dar es Salaam called Sinza Kwa Remmy, named after the musician when he moved to the area in the 1980s. — BBCOnline.